Bulk Data Movement with Fedena

Data and data analysis plays a vital role in our day to day life. Everything we look into is associated with some kind of data or others. Data plays a key role in arriving at any decisions. It supports in decision making. In short, data can be defined as information that has been translated into any form that is more convenient to move or process.

Data can be different. It can be either meaningful or meaningless or can be abstract. When it comes to computing, data is the prime component. Every aspect related to computing is associated with some kind of data processing. Conversion of computing problems into executable programs involves analysis, understanding of problem data. Object-Oriented Programming makes this conversion much easier by connecting all real-world things to objects that have data fields (attributes that describe the object) and associated procedures are known as methods.

In Fedena we manage gbs of institution data.  In between Fedena’s evolution, we use to get plenty of requests related to data retrieval and data entry. So data management has always been a topic of study. So for the bulk movement of data into and from Fedena. Considering the importance of data processing, what we could do is create an exporting and importing interface. But fedena expands rapidly, it’s not wise to implement this for every new module made. So what we needed is an engine to handle data bulk export and import, which can adapt to the changes in Fedena easily. Hence we design Data Export and Custom Import plugins which handle bulk import and export of data in a simple manner.

And for the engine design, we exploited the MVC architecture. In MVC architecture the data and logic are separated into different layers. The model represents the physical data in your application. MVC architecture converts the database into a model structure with the help of some object-oriented principles. Model layer helps in the creation and updating of data inside each table. The data can be captured in the form of attributes and their values of objects. Each table inside the database will be mapped to one model. We can add more power to the data analysis by creating relationships between data in different tables.

Two engines one handles import and other export. In both, we define a particular schema for data for each model. When importing according to the schema the engine creates a CSV file which defines headers for the particular model. User downloads it and merges with the data they want to import. The header as I said will be defined by the engine, and the header for a user selected model will be closely associated with the real world object itself. For example, a header for Student model will contain first name, last name, admission number, address, batch name, etc. While during data export the same schema is set for each model. The user selects the model, accordingly, the engine creates export files. The engines make it easier to extend the functionalities for a new module within Fedena.

When it comes to ERP applications data movement has an important role. Almost all ERP applications use an extensive set of data for its functioning. The way each of the application process and use data may differ. Fedena custom import and export plugin helps a great deal in data management. Fedena custom import module is primarily developed to move bulk data into fedena. This eases the time-consuming data entry process. Fedena custom import plugin supports commonly seen file formats which user can choose as per the requirement. In overall complexity in the data upload is greatly reduced.

Fedena data export module supports the user to export Model data for various purposes. There will be a predefined set of exports (model) available. Model data can be downloaded in either of formats XML or CSV. These exported data are of various use. The export operation serves many purposes like the backup of a certain model, data transfer to some other application, for acquiring meaningful insights from the dataset, data-driven decision making, etc. And overall today’s business, organizational process plannings are based on data. Data is constantly analyzed to for evaluation. Different kind of outputs is made out of the same set of data. Data exported from Fedena can be used for many of all of those reasons.

This article is written by:

Priyanka TP

Somebody who literally cooks code, Priyanka is kind of girl who works smart in kitchen and on computer with the same zeal. And her codes are equally noteworthy as her delicious recipes.