Real-Time and Big Data GIS: Best Practices

This session started off with a discussion and review of all the system requirements. I could type up the specs.... but why? Follow the Esri docs, it's all listed here:

Spatiotemporal Big Data Store:

The recommendation for having multiple instances or nodes of the Big Data Store (BDS) for production systems was a point that was driven home during this session. The reason behind this is that having multiple nodes makes for a redundant more highly available system. The BDS by default has auto-rebalancing of data upon node membership that changes when you add or remove a new BDS node or in other words the BDS is self-healing with multiple nodes. If one goes down, it is repaired by the replicas that are spread out across other nodes. This is a great feature that’s available and doesn’t require any additional licensing and prevents data loss and downtime.

As mentioned in the docs Esri is really pushing to ensure people know that the Spatiotemporal BDS should be installed on its own machine as it will consume 50% of resources of whatever machine it is on. So much so that at 10.7 when you install the BDS, there will be a notification ensuring that you are installing the BDS on its own server. If the BDS should run out of disk space because of increased data volumes, failure to monitor disk space or not having properly set your data retention policies on your services the BDS will go to Read Only mode. It will still save and maintain the data stored in your BDS, however being in Read Only mode means it will stop writing new data until additional resources are added to the machine.

So, what’s coming down the pipeline at 10.7 for the BDS you ask? At 10.7 there will be a Batch exports option available to export data from the BDS to Azure Blob or Amazon S3 and will also be able to export by script, however this data will only be available to export as a csv.

Things to Note – GeoEvent and Beyond:

There are 2 new GeoEvent patches that address a whole host of issues for GeoEvent 10.6 and 10.6.1. If you have a client running GeoEvent at these versions Esri highly recommends getting these patches in place.

  • Due to the changes Esri is implementing regarding TLS 1.2 GeoEvent Server 10.3.1 will no longer be able to connect to AGOL after update in April.
  • GeoEvent by default is only allocated 4gb of RAM for the Java virtual Machine (JVM)
    • If utilizing more, you must go into config files and update this amount.
  • GeoEvent Server by default is only able to maintain the state of 1000 unique Track_IDs
    • This value can be increased by editing a config.
    • "/etc/com.esri.ges.manager.servicemanager.cfg"
  • It is more performant to create multiple single GeoEvent Services then to have one branching out to multiple outputs.
  • Look at adjusting global settings to reduce the size of the WebSocket coming through. This can increase performance.
  • Esri doesn’t generally recommend Federating the GeoEvent Real time Server.
  • GeoEvent connection to AGOL as a Datastore: Writing a new service to this location is not supported. You can access your services but not publish to this location.
  • New video series coming out in 2019 addressing 3 topics – coming soon from Esri!
    • Fundamentals
    • Workflows
    • Tips and Tricks
Stay tuned for more updates on Esri #DevSummit2019 on www.gisinc.com/blog.