ProductIds = await Common.GetProductIds(configuration.GetConnectionString("DataConnection")) Īwait Common.TruncateLoadingTables(configuration.GetConnectionString("DataConnection")) IServiceProvider serviceProvider = serviceCollection.BuildServiceProvider() ĭateTime processingDateTime = DateTime.Now ServiceCollection serviceCollection = new ServiceCollection() Public static IConfigurationRoot configuration
Prepare a DataSet with all desired DataTables, disable constraints from those tables to allow faster inserting, fire up threads to insert into them and then finally insert and merge the results. With the project based on that linked to in my prior post above I did the following to store the returned data in memory and do a single bulk insert into SQL after the data had been gathered. I got around this by having each thread take an exclusive lock on the DataTable when adding a row which isn’t the optimal solution but was good enough for my use case. I did however experience problems inserting rows into my in-memory DataTable which I was using the returned data before the bulk insert as it turns out DataTable is not thread safe and I was getting DataTable internal index is corrupted errors frequently when trying to add rows. In order to get around this I decided to store the results returned in memory and then do a single bulk insert when all the data had been gathered, although it’s not the best idea to store a large amount of data in memory before comitting it the amount of data returned wasn’t enough to cause any problems. This worked fine when a low number of threads was used but then started to experience long delays due to the sheer number of connections being opened and closed to the SQL server and the number of rows being written. As part of a project where I was running multiple threads making API calls like this I was making a stored procedure call to log the results to an SQL database for each result returned.
0 kommentar(er)
