PEAK TO PEAK.
HIGH PERFORMANCE WITH LOBSTER_DATA.
Using the programming language Java in combination with a modern system architecture make Lobster_data a high performance enterprise solution with a whole range of technical advantages. Our innovative approach splits the processing and communication of data packets to ensure secure transmission.
CUSTOMERS, WHO RELY ON LOBSTER.
BENEFITS.
Lobster is committed to Green IT and minimises the environmental footprint of your data processing. We avoid queues by transferring files in batches instead of individually and ensure adequate management of many short transactions. This not only reduces latency and processing time, but also saves costs on expensive transmission methods through compression and automation.
Let’s optimise your data processing together!
Multithreading
Lobster_data supports multithreading, so that both the simultaneous execution of several threads in a programme and – through the use of several CPU cores – a significant improvement in performance are possible.
EFFICIENT MEMORY MANAGEMENT
Lobster_data features an automatic, efficient memory management system through its garbage collection function, which significantly reduces the risk of memory leaks and crashes.
JUST-IN-TIME COMPILATION (JIT)
JIT compilation in Lobster_data improves performance by allowing the JVM to dynamically compile bytecode into native machine code at runtime.
PLATFORM INDEPENDENCE
As a Java-based software, Lobster_data can be executed on any platform where a Java Virtual Machine (JVM) is installed, providing platform independence.
THOUSANDS OF AS400 REQUESTS CAN BE EXECUTED IN MINUTES INSTEAD OF HOURS
x Hide
COMPONENTS PERFORMANCE & FUNCTIONAL OVERVIEW.
Direct establishment of TCP connections in case the devices and machines to be connected are not MQTT and/or OPC UA capable.
PROCESS LOAD OPTIMISATION
The Lobster_data module "Process Load Optimisation" (PLO) offers optimised processing at very high process volumes.
ASYNCHRONOUS SENDING MODULE
The ASM add-on module bundles the sending of any number of files to communication partners for a previously defined point in time.
XML Streaming
The use of XML streaming enables extremely fast processing of large XML files (in the range of multiple GB) with minimal memory requirements.
With the press of a button we can refer back to our most-used formats. Tasks that once took days, are now completed in just a few hours. All thanks to Lobster_data.
Bardo Schütz
Director of Innovation and Technology (CTO) at Customs Support Group, Business Universiteit, Wassenaar, Netherlands
- Tips & Tricks
- Load balancing
- ETL/ELT
- Use Case
Tips & Tricks
Hardware
Lobster_data offers tips and tricks for improving overall performance in addition to server hardware upgrades.
Performance analysis
During operation, performance analyses can be used to make targeted optimisations based on statistical data. Lobster_data provides dedicated hints for optimisations through workload monitoring and the listing of APIs with the most conversions, errors, data volumes and longest processing times.
Load balancing
With Lobster, the node controller acts as an internal load balancer and offers several performance benefits.
IMPROVED AVAILABILITY AND UPTIME
By distributing the incoming data traffic to several working nodes, it is ensured that if one node fails, the traffic is redirected to another node that is available. If the node controller fails, the next working node automatically takes over its task. No downtime of the overall system occurs as maintenance work or updates can be carried out successively per node.
FASTER RESPONSE TIMES
Load balancing shortens response times by giving connections to nodes that are less busy or optimised for specific tasks.
SCALABILITY
The load balancer enables handling of more data traffic and API requests by distributing the load across multiple nodes, allowing for better scalability and capacity planning. If data traffic increases, additional nodes can be added to the pool to handle the load.
BETTER RESOURCE UTILISATION
The load balancer ensures optimal utilisation of all nodes in the pool, minimising both the over and underutilisation of resources and improving the overall efficiency of the system.
ETL/ELT
ETL (Extract, Transform, Load) and ELT (Extract, Load, Transform) are data integration processes used to move and transform data from one or more source systems to a target system. The ETL/ELT module in Lobster_data supports the merging and processing of large amounts of data from different data sources.
High-speed ETL/ELT processes can be used to load data into a target database (data warehouse, data lake). With the ETL/ELT module, the high-performance processing of all data cleansing and transformation rules is also carried out and thus the data is quickly available in the target system. Existing interfaces to databases, ERP, logistics or shop systems can be used quickly and with little effort for the implementation of ETL/ELT. The Lobster ETL/ELT module offers high-performance DWH pre-processing of mass data for files up to 6 GB in size. The time required for data integration and data provision in the context of business intelligence is therefore significantly reduced.
Use Case
Lobster_data is high performing. For example, with 2 TB of data volume and 22 million data requests, a server with 16 cores and 48 GB of main memory usually has a load of only 30%. Nevertheless, for very large data volumes, there is the possibility of additional distribution, so that the central node controller splits the incoming data traffic among one or more working nodes in different systems and prevents overloading of the individual system.
When a request is sent to a node controller, it uses predefined settings to determine which working node in the cluster should handle the request. Once a node is selected, the load balancing system forwards the request directly to that node.
