Skip to content

Cannot import database in low-ram profile #670

New issue
New issue
Closed
Closed
Cannot import database in low-ram profile#670
Assignees
lvca
Labels
questionFurther information is requested

Description

@retaggi
retaggi
opened on Dec 14, 2022

ArcadeDB Version: v22.12.1-SNAPSHOT

JDK Version: openjdk 11.0.17 2022-10-18

OS: Ubuntu 22.04.1 LTS

Expected behavior

Even with low-ram profile (128mb ram or lower) I expect to import any database if I have enough space on my disk.

Actual behavior

java.lang.OutOfMemoryError (java heap space)

`
2022-12-14 10:50:15.027 INFO [ArcadeDBServer] <ArcadeDB_0> ArcadeDB Server v22.12.1-SNAPSHOT (build 81566f3/1670890687610/main) is starting up...
2022-12-14 10:50:15.029 INFO [ArcadeDBServer] <ArcadeDB_0> Starting ArcadeDB Server with plugins [Redis, MongoDB, Postgres, GremlinServer] ...
2022-12-14 10:50:15.051 INFO [ArcadeDBServer] <ArcadeDB_0> - JMX Metrics Started...
2022-12-14 10:50:15.090 INFO [ServerSecurity] <ArcadeDB_0> Creating root user with the provided password
2022-12-14 10:50:15.224 INFO [ArcadeDBServer] <ArcadeDB_0> Creating default database 'Imported'...
2022-12-14 10:50:15.391 INFO [SourceDiscovery] <ArcadeDB_0> Analyzing url: https://github.com/ArcadeData/arcadedb-datasets/raw/main/orientdb/OpenBeer.gz...
2022-12-14 10:50:16.343 INFO [SourceDiscovery] <ArcadeDB_0> Recognized format OrientDB (parsingLimitBytes=9.54MB parsingLimitEntries=0)
Importing OrientDB database from stream to './databases/Imported'
Creation of the schema: types, properties and indexes

  • Created type 'V' with the following properties {}
  • Created type 'Beer' with the following properties {id=INTEGER}
  • Created type 'Brewery' with the following properties {id=INTEGER}
  • Created type 'Category' with the following properties {id=INTEGER}
  • Created type 'E' with the following properties {}
  • Created type 'HasBrewery' with the following properties {}
  • Created type 'HasCategory' with the following properties {}
  • Created type 'HasStyle' with the following properties {}
  • Created type 'Style' with the following properties {id=INTEGER}
  • Created index UNIQUE on Beer[id]
  • Created index UNIQUE on Brewery[id]
  • Created index UNIQUE on Style[id]
    java.lang.OutOfMemoryError: Java heap space
    Dumping heap to java_pid1.hprof ...
    Heap dump file created [81224816 bytes in 0.106 secs]
  • Status update: parsed 0 (0/sec) - 0 documents (0/sec) - 0 vertices (0/sec) - 0 edges (0/sec) - 0 skipped edges - 0 linked edges (0/sec - 0%)
    2022-12-14 10:50:16.344 INFO [Importer] <ArcadeDB_0> Checking schema...Exception in thread "main" com.arcadedb.exception.CommandExecutionException: Error on importing database
    at com.arcadedb.query.sql.parser.ImportDatabaseStatement.executeSimple(ImportDatabaseStatement.java:70)
    at com.arcadedb.query.sql.executor.SingleOpExecutionPlan.executeInternal(SingleOpExecutionPlan.java:115)
    at com.arcadedb.query.sql.parser.SimpleExecStatement.execute(SimpleExecStatement.java:56)
    at com.arcadedb.query.sql.parser.Statement.execute(Statement.java:82)
    at com.arcadedb.query.sql.parser.Statement.execute(Statement.java:66)
    at com.arcadedb.query.sql.SQLQueryEngine.command(SQLQueryEngine.java:88)
    at com.arcadedb.database.EmbeddedDatabase.command(EmbeddedDatabase.java:1223)
    at com.arcadedb.server.ArcadeDBServer.loadDefaultDatabases(ArcadeDBServer.java:468)
    at com.arcadedb.server.ArcadeDBServer.start(ArcadeDBServer.java:130)
    at com.arcadedb.server.ArcadeDBServer.main(ArcadeDBServer.java:91)
    Caused by: com.arcadedb.integration.importer.ImportException: Error on parsing source 'https://github.com/ArcadeData/arcadedb-datasets/raw/main/orientdb/OpenBeer.gz (compressed=true size=1085781)'
    at com.arcadedb.integration.importer.Importer.load(Importer.java:62)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at com.arcadedb.query.sql.parser.ImportDatabaseStatement.executeSimple(ImportDatabaseStatement.java:65)
    ... 9 more
    Caused by: com.arcadedb.index.IndexException: Error on creating index on type 'Category', properties [id]
    at com.arcadedb.schema.EmbeddedSchema.createTypeIndex(EmbeddedSchema.java:506)
    at com.arcadedb.schema.EmbeddedSchema.getOrCreateTypeIndex(EmbeddedSchema.java:541)
    at com.arcadedb.integration.importer.OrientDBImporter.parseIndexes(OrientDBImporter.java:772)
    at com.arcadedb.integration.importer.OrientDBImporter.createRecord(OrientDBImporter.java:424)
    at com.arcadedb.integration.importer.OrientDBImporter.createRecords(OrientDBImporter.java:384)
    at com.arcadedb.integration.importer.OrientDBImporter.parseRecords(OrientDBImporter.java:333)
    at com.arcadedb.integration.importer.OrientDBImporter.parseInputFile(OrientDBImporter.java:286)
    at com.arcadedb.integration.importer.OrientDBImporter.run(OrientDBImporter.java:175)
    at com.arcadedb.integration.importer.format.OrientDBImporterFormat.load(OrientDBImporterFormat.java:48)
    at com.arcadedb.integration.importer.Importer.loadFromSource(Importer.java:90)
    at com.arcadedb.integration.importer.Importer.load(Importer.java:53)
    ... 14 more
    Caused by: com.arcadedb.exception.DatabaseOperationException: Error during read lock
    at com.arcadedb.database.EmbeddedDatabase.executeInReadLock(EmbeddedDatabase.java:1357)
    at com.arcadedb.database.EmbeddedDatabase.commit(EmbeddedDatabase.java:359)
    at com.arcadedb.database.EmbeddedDatabase.transaction(EmbeddedDatabase.java:998)
    at com.arcadedb.schema.EmbeddedSchema.lambda$createTypeIndex$4(EmbeddedSchema.java:484)
    at com.arcadedb.database.EmbeddedDatabase.executeInWriteLock(EmbeddedDatabase.java:1372)
    at com.arcadedb.database.EmbeddedDatabase.recordFileChanges(EmbeddedDatabase.java:1392)
    at com.arcadedb.schema.EmbeddedSchema.recordFileChanges(EmbeddedSchema.java:1370)
    at com.arcadedb.schema.EmbeddedSchema.createTypeIndex(EmbeddedSchema.java:483)
    ... 24 more
    Caused by: java.lang.OutOfMemoryError: Java heap space
    at com.arcadedb.database.Binary.(Binary.java:64)
    at com.arcadedb.database.Binary.(Binary.java:71)
    at com.arcadedb.engine.WALFile.writeTransactionToBuffer(WALFile.java:237)
    at com.arcadedb.engine.TransactionManager.createTransactionBuffer(TransactionManager.java:138)
    at com.arcadedb.database.TransactionContext.commit1stPhase(TransactionContext.java:507)
    at com.arcadedb.database.TransactionContext.commit(TransactionContext.java:114)
    at com.arcadedb.database.EmbeddedDatabase.lambda$commit$2(EmbeddedDatabase.java:364)
    at com.arcadedb.database.EmbeddedDatabase$$Lambda$111/0x0000000840128c40.call(Unknown Source)
    at com.arcadedb.database.EmbeddedDatabase.executeInReadLock(EmbeddedDatabase.java:1346)
    at com.arcadedb.database.EmbeddedDatabase.commit(EmbeddedDatabase.java:359)
    at com.arcadedb.database.EmbeddedDatabase.transaction(EmbeddedDatabase.java:998)
    at com.arcadedb.schema.EmbeddedSchema.lambda$createTypeIndex$4(EmbeddedSchema.java:484)
    at com.arcadedb.schema.EmbeddedSchema$$Lambda$156/0x0000000840207840.call(Unknown Source)
    at com.arcadedb.database.EmbeddedDatabase.executeInWriteLock(EmbeddedDatabase.java:1372)
    at com.arcadedb.database.EmbeddedDatabase.recordFileChanges(EmbeddedDatabase.java:1392)
    at com.arcadedb.schema.EmbeddedSchema.recordFileChanges(EmbeddedSchema.java:1370)
    at com.arcadedb.schema.EmbeddedSchema.createTypeIndex(EmbeddedSchema.java:483)
    at com.arcadedb.schema.EmbeddedSchema.getOrCreateTypeIndex(EmbeddedSchema.java:541)
    at com.arcadedb.integration.importer.OrientDBImporter.parseIndexes(OrientDBImporter.java:772)
    at com.arcadedb.integration.importer.OrientDBImporter.createRecord(OrientDBImporter.java:424)
    at com.arcadedb.integration.importer.OrientDBImporter.createRecords(OrientDBImporter.java:384)
    at com.arcadedb.integration.importer.OrientDBImporter.parseRecords(OrientDBImporter.java:333)
    at com.arcadedb.integration.importer.OrientDBImporter.parseInputFile(OrientDBImporter.java:286)
    at com.arcadedb.integration.importer.OrientDBImporter.run(OrientDBImporter.java:175)
    at com.arcadedb.integration.importer.format.OrientDBImporterFormat.load(OrientDBImporterFormat.java:48)
    at com.arcadedb.integration.importer.Importer.loadFromSource(Importer.java:90)
    at com.arcadedb.integration.importer.Importer.load(Importer.java:53)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at com.arcadedb.query.sql.parser.ImportDatabaseStatement.executeSimple(ImportDatabaseStatement.java:65)
    `

Steps to reproduce

docker run -d --rm  -p 2480:2480 -p 2424:2424 -p 6379:6379 -p 5432:5432 -p 8182:8182 \
--name arcadedb \
--env ARCADEDB_OPTS_MEMORY="-Xms128M -Xmx128M" \
--env JAVA_OPTS="-Darcadedb.server.rootPassword=playwithdata \
-Darcadedb.profile=low-ram \
-Darcadedb.server.defaultDatabases=Imported[root]{import:https://github.com/ArcadeData/arcadedb-datasets/raw/main/orientdb/OpenBeer.gz} \
-Darcadedb.server.plugins=Redis:com.arcadedb.redis.RedisProtocolPlugin,MongoDB:com.arcadedb.mongo.MongoDBProtocolPlugin,Postgres:com.arcadedb.postgres.PostgresProtocolPlugin,GremlinServer:com.arcadedb.server.gremlin.GremlinServerPlugin " \
arcadedata/arcadedb:latest

I tried with some tuning like the following, without success:

-Darcadedb.sqlStatementCache=10 \
-Darcadedb.freePageRAM=100 \
-Darcadedb.indexCompactionRAM=1 \
-Darcadedb.maxPageRAM=1024 \

I tried even without defaultDatabases and with import database https://github.com/ArcadeData/arcadedb-datasets/raw/main/orientdb/OpenBeer.gz after the startup of the server.

Thank you

Activity

  1. retaggi
    changed the title [-]Cannot import database in low-profile mode[/-] [+]Cannot import database in low-ram profile[/+] on Dec 14, 2022
  2. lvca
    self-assigned this
    on Dec 14, 2022
  3. lvca
    added
    questionFurther information is requested
    on Dec 14, 2022

  4. lvca commented on Dec 14, 2022

    @lvca
    lvca
    on Dec 14, 2022
    Member

    With such low RAM, please try with commitEvery equals to 50 or less:

    https://docs.arcadedb.com/#_additional-settings

  5. retaggi commented on Dec 15, 2022

    @retaggi
    retaggi
    on Dec 15, 2022
    Author

    i tried with

    IMPORT DATABASE https://github.com/ArcadeData/arcadedb-datasets/raw/main/orientdb/OpenBeer.gz
with commitEvery = 50

    and

    IMPORT DATABASE https://github.com/ArcadeData/arcadedb-datasets/raw/main/orientdb/OpenBeer.gz
with commitEvery = 1

    with the same issue

    2022-12-15 08:09:03.402 INFO  [SourceDiscovery] <ArcadeDB_0> Analyzing url: https://github.com/ArcadeData/arcadedb-datasets/raw/main/orientdb/OpenBeer.gz...
2022-12-15 08:09:03.884 INFO  [SourceDiscovery] <ArcadeDB_0> Recognized format OrientDB (parsingLimitBytes=9.54MB parsingLimitEntries=0)
Importing OrientDB database from stream to './databases/test'
Creation of the schema: types, properties and indexes
- Created index UNIQUE on Beer[id]
- Created index UNIQUE on Brewery[id]
- Created index UNIQUE on Style[id]
- Created index UNIQUE on Category[id]
Creation of records started: creating vertices and documents records (edges on the next phase)
- Status update: parsed 9,997 (9,997/sec) - 0 documents (0/sec) - 5,417 vertices (5,417/sec) - 0 edges (0/sec) - 0 skipped edges - 0 linked edges (0/sec - 0%)
2022-12-15 08:09:03.885 INFO  [Importer] <ArcadeDB_0> Checking schema...<ArcadeDB_0> Error on command execution (PostCommandHandler)
com.arcadedb.exception.CommandExecutionException: Error on importing database
        at com.arcadedb.query.sql.parser.ImportDatabaseStatement.executeSimple(ImportDatabaseStatement.java:70)
        at com.arcadedb.query.sql.executor.SingleOpExecutionPlan.executeInternal(SingleOpExecutionPlan.java:115)
        at com.arcadedb.query.sql.executor.ScriptLineStep.syncPull(ScriptLineStep.java:53)
        at com.arcadedb.query.sql.executor.ScriptExecutionPlan.doExecute(ScriptExecutionPlan.java:97)
        at com.arcadedb.query.sql.executor.ScriptExecutionPlan.fetchNext(ScriptExecutionPlan.java:57)
        at com.arcadedb.query.sql.parser.LocalResultSet.fetchNext(LocalResultSet.java:46)
        at com.arcadedb.query.sql.parser.LocalResultSet.<init>(LocalResultSet.java:40)
        at com.arcadedb.database.EmbeddedDatabase.executeInternal(EmbeddedDatabase.java:1296)
        at com.arcadedb.database.EmbeddedDatabase.execute(EmbeddedDatabase.java:1244)
        at com.arcadedb.server.ServerDatabase.execute(ServerDatabase.java:415)
        at com.arcadedb.server.http.handler.PostCommandHandler.executeScript(PostCommandHandler.java:100)
        at com.arcadedb.server.http.handler.PostCommandHandler.execute(PostCommandHandler.java:71)
        at com.arcadedb.server.http.handler.DatabaseAbstractHandler.execute(DatabaseAbstractHandler.java:85)
        at com.arcadedb.server.http.handler.AbstractHandler.handleRequest(AbstractHandler.java:109)
        at io.undertow.server.RoutingHandler.handleRequest(RoutingHandler.java:93)
        at io.undertow.server.handlers.PathHandler.handleRequest(PathHandler.java:104)
        at io.undertow.server.Connectors.executeRootHandler(Connectors.java:391)
        at io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:256)
        at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136)
        at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:59)
        at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
        at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
        at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
        at org.xnio.nio.WorkerThread.run(WorkerThread.java:591)
Caused by: com.arcadedb.integration.importer.ImportException: Error on parsing source 'https://github.com/ArcadeData/arcadedb-datasets/raw/main/orientdb/OpenBeer.gz (compressed=true size=1085781)'
        at com.arcadedb.integration.importer.Importer.load(Importer.java:62)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at com.arcadedb.query.sql.parser.ImportDatabaseStatement.executeSimple(ImportDatabaseStatement.java:65)
        ... 23 more
Caused by: com.arcadedb.exception.TransactionException: Transaction error on commit
        at com.arcadedb.database.TransactionContext.commit1stPhase(TransactionContext.java:518)
        at com.arcadedb.database.TransactionContext.commit(TransactionContext.java:114)
        at com.arcadedb.database.EmbeddedDatabase.lambda$commit$2(EmbeddedDatabase.java:364)
        at com.arcadedb.database.EmbeddedDatabase.executeInReadLock(EmbeddedDatabase.java:1346)
        at com.arcadedb.database.EmbeddedDatabase.commit(EmbeddedDatabase.java:359)
        at com.arcadedb.integration.importer.OrientDBImporter.parseRecords(OrientDBImporter.java:340)
        at com.arcadedb.integration.importer.OrientDBImporter.parseInputFile(OrientDBImporter.java:286)
        at com.arcadedb.integration.importer.OrientDBImporter.run(OrientDBImporter.java:175)
        at com.arcadedb.integration.importer.format.OrientDBImporterFormat.load(OrientDBImporterFormat.java:48)
        at com.arcadedb.integration.importer.Importer.loadFromSource(Importer.java:90)
        at com.arcadedb.integration.importer.Importer.load(Importer.java:53)
        ... 28 more
Caused by: com.arcadedb.exception.ArcadeDBException: Error in execution in lock
        at com.arcadedb.utility.RWLockContext.executeInReadLock(RWLockContext.java:58)
        at com.arcadedb.index.lsm.LSMTreeIndex.put(LSMTreeIndex.java:391)
        at com.arcadedb.database.TransactionIndexContext.commit(TransactionIndexContext.java:191)
        at com.arcadedb.database.TransactionContext.commit1stPhase(TransactionContext.java:474)
        ... 38 more
Caused by: java.lang.OutOfMemoryError: Java heap space

    I think that the problem is that, before the parsing of the data, the OpenBeer.gz is fully loaded in RAM and if there is no space, it just raise the OutOfMemoryError exception. I think its missing some logic to chunk the OpenBeer.gz

  6. lvca commented on Dec 15, 2022

    @lvca
    lvca
    on Dec 15, 2022
    Member

    This is the main loop:

    database.begin();

    while (reader.peek() == BEGIN_OBJECT) {
      final Map<String, Object> attributes = parseRecord(reader, false);

      final String className = (String) attributes.get("@class");

      if (phase == PHASE.CREATE_SCHEMA || phase == PHASE.CREATE_RECORDS)
        createRecords(processedItems, attributes, className);
      else
        createEdges(processedItems, attributes, className);

      ++processedItems;

      if (processedItems > 0 && processedItems % batchSize == 0) {
        database.commit();
        database.begin();
      }

      if (reader.peek() == NULL)
        // FIX A BUG ON ORIENTDB EXPORTER WHEN GENERATE AN EMPTY RECORD
        reader.skipValue();
    }

    database.commit();

    The input file is read byte per byte and as soon as a record is completed it's imported. So the whole file is not read in RAM first. How much heap memory to the JVM?

  7. lvca commented on Jan 4, 2023

    @lvca
    lvca
    on Jan 4, 2023
    Member

    Closing the issue because no answer was received.

  8. lvca
    closed this as completedon Jan 4, 2023
  9. mergify
    added a commit that references this issue on Feb 8, 2026

    chore(deps-dev): bump org.codehaus.mojo:license-maven-plugin from 2.4…

    e83f28e
  10. tae898
    added a commit that references this issue on Jun 28, 2026

    chore(deps-dev): bump org.codehaus.mojo:license-maven-plugin from 2.4…

    5aeb5a6
  11. mergify
    added a commit that references this issue on Jul 19, 2026

    chore(deps): bump io.github.jbellis:jvector from 4.0.0-rc.8-hf1 to 4.…

    b662699
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

questionFurther information is requested

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions