sqly-me

Project Info

Porretas thumbnail

Project Description


Read the README description at the repository to better understand how it works.

However, a simple description is: When there is a need of using more than one dataset, there is a need to cross them at some point. This project is a Python Module capable of doing such merge automatically. It is an enabling tool to allow data scientists (for instance) to avoid the effort of doing it. When it merges all the structured data the user wants, it returns a report and a sqlite database file for the user to then proceed on their analysis.

This project proposes a solution for multiple challenges by creating a Python module capable of taking multiple .csv or .xslx files and combining them into one single database, on a sqlite file. In addition, it provides a report with all files that were able to merge into such database.
It is an enabling tool, not a data analysis and information giving software. However, it is capable of checking all possible combinations inside the total range of files given. For instance, if 10 files are given and two different clusters of data exists on them, it will generate 2 different databases alongside with their respective reports.


Data Story


All outcomes from the data used on this project is present at the Google Drive folder:
https://drive.google.com/drive/folders/13G5ToX0MXUi_SUyzVlTOr6DN-btfo4ws?usp=sharing


Evidence of Work

Video

Homepage

High-Res Image

Team DataSets

Digital Culture

Description of Use: For this challenge, multiple datasets from this URL were used. Which ones were used and which ones successfully merged are presented at the report on the Google Drive Folder: https://drive.google.com/drive/folders/13G5ToX0MXUi_SUyzVlTOr6DN-btfo4ws

Data Set

Physical Culture

Description of Use: For this challenge, multiple datasets from this URL were used. Which ones were used and which ones successfully merged are presented at the report on the Google Drive Folder: https://drive.google.com/drive/folders/13G5ToX0MXUi_SUyzVlTOr6DN-btfo4ws

Data Set

Households in Housing Stress

Description of Use: Investing in Regions datasets used. Showcasing our Regions datasets. In addition to this one, these following datasets were used: https://data.sa.gov.au/data/dataset/households-in-25-housing-stress https://data.sa.gov.au/data/dataset/housing-stress-50-of-income The outcome is displayed at the team google drive folder: https://drive.google.com/drive/folders/13G5ToX0MXUi_SUyzVlTOr6DN-btfo4ws

Data Set

TM-Link Data Discovery

Description of Use: We crossed both TM-Link's application and applicants dataset to get an overview of the given data. The outcome is a report giving an analysis of that merge. However, the real outcome on the project is the capability of crossing any other dataset, as the user prefers. The report is found on the following link: https://drive.google.com/drive/folders/13G5ToX0MXUi_SUyzVlTOr6DN-btfo4ws?usp=sharing

Data Set

Small Area Labour Markets (SALM) Data

Description of Use: Small Area Labour Markets (SALM) Data redirects to the following page: https://employment.gov.au/small-area-labour-markets-publication From that page, two different .csv files (SA2 and LGA data tables) were downloaded and used on the project. The outcome is a report and a sqlite database file containing the united data of those and provided on the Google Drive folder, on the team details

Data Set

Challenges

Investing in Regions

How might the Government prioritise its investment in regional South Australia for greatest regional development benefit?

Go to Challenge | 10 teams have entered this challenge across all checkpoints that have passed.

Showcasing our regions

How might we promote South Australian regions to boost regional development?

Go to Challenge | 16 teams have entered this challenge across all checkpoints that have passed.

Thrive or survive: how can we adapt for the future?

What will Australia in 2050 look like?

Go to Challenge | 38 teams have entered this challenge across all checkpoints that have passed.

Digital Culture

How do we make our digital cultural heritage collections engaging for online audiences? What experiences should we be developing beyond the search and retrieve box to visualise gallery, library and museum collections online and encourage their reuse and good storytelling?

Go to Challenge | 14 teams have entered this challenge across all checkpoints that have passed.

Physical Culture

How might we better integrate our digital collections and datasets into our physical gallery, library, or museum spaces?

Go to Challenge | 9 teams have entered this challenge across all checkpoints that have passed.

Australia@Sea: what is our future relationship with the ocean environment?

Our oceans are vital to the world’s economy and provide services for all Australians including food security, industries, tourism, and well-being.

Go to Challenge | 17 teams have entered this challenge across all checkpoints that have passed.

Australia’s Future Employment

Choose one of the following questions to address: 1. How can recent and future changes in the labour market help prepare young people for job opportunities? 2. What can we learn from case studies of regional labour markets? For example, what does rapid change in the industries or occupations within a region tell us about the needs of employers/workers in other regional labour markets

Go to Challenge | 38 teams have entered this challenge across all checkpoints that have passed.

TM-Link Data Discovery

TM-Link is a newly available trade mark database developed in collaboration between IP Australia, Swinburne University and Melbourne University. TM-Link includes administrative data from jurisdictions across the world, linked at the application level by advanced neural network algorithms. We are encouraging hackers to explore this new data set and consider what creative visualisations, innovative insights and/or opportunities to further enrich the data they might imagine.

Go to Challenge | 8 teams have entered this challenge across all checkpoints that have passed.

Local Government Information Technology Association of South Australia

How might we identify opportunities for improvement or new Council services, infrastructure and facilities to benefit community outcomes in South Australia?

Go to Challenge | 15 teams have entered this challenge across all checkpoints that have passed.

Helping a social impact ‘start up’ (small organisation) to tell their story

Small and informal community/interest groups who have formed to solve local problems need data to know if their activities are making a difference and to re-design programs. How can we help these groups tell their story through data so they can seek support (political, financial, and on the ground) by showing how their programs are working, and decide where to focus next?

Go to Challenge | 23 teams have entered this challenge across all checkpoints that have passed.

Back to Projects