The process of getting your data reliably into Socrata is one of the most important parts of any open data project. Given the amount and types of systems open data is pulled from, it’s absolutely no wonder this is can be a daunting task. Once you gather and then transform the data, you know you still have to upload the data into Socrata.
Within Socrata, the Truth and Ingress engineering team is responsible for making sure all data is reliably placed into the system. Of course, nobody would argue that this is an easy software solution. In fact, data ingress or the process of uploading data into Socrata is one of the hardest and most important areas for any open data publisher. The faster and more reliably they can get data out of back end systems and into Socrata, the faster the data is available.
To help explain the process, we asked John Kew a Socrata engineer from the Truth and Ingress team to provide a quick technical overview of the data ingress process.