database documentation updating
This commit is contained in:
@@ -1,102 +1,89 @@
|
|||||||
# Original Version (by bram)
|
# Original Database - Websocket connection Version (by bram)
|
||||||
|
For our project, we needed to establish an efficient and functional connection between a live data collector and a database where this data would be stored.
|
||||||
|
|
||||||
|
The data we collected originated from live "nodes" which were small boxes equipped with various types of sensors to gather specific data from their surroundings.
|
||||||
|
|
||||||
|
This collected data needed to be transmitted to a database to facilitate its presentation on a website we were developing. The website would retrieve the data from the database and display it in various formats.
|
||||||
|
|
||||||
|
Given the critical nature of this data connection for our project, it was imperative that it functioned reliably. Bram was tasked with designing a connection in Python between the WebSocket (live server) and the database (data storage).
|
||||||
|
|
||||||
|
Since we had the WebSocket data on a Raspberry Pi, it made sense to implement the connection on the Pi itself. This presented an opportunity for Bram to acquire knowledge about Python, considering he initially lacked experience with this programming language.
|
||||||
## Python code + explaination
|
## Python code + explaination
|
||||||
|
In the given Raspberry Pi, a file named "data.py" was created, from which this script could be called when needed.
|
||||||
|
|
||||||
We wanted to make a working connection between our websocket wich runs all the data gatherd by our nodes and a live feed to our database.
|
At the beginning of the file, the code starts with importing the different required libraries.
|
||||||
So we set out to make this connection using python.
|
```py
|
||||||
|
|
||||||
At first we needed to import the folowing librarys:
|
|
||||||
|
|
||||||
```js
|
|
||||||
// everything is running async so the code can run together and not interfere with eachother.
|
|
||||||
import asyncio
|
import asyncio
|
||||||
import websockets
|
import websockets
|
||||||
// a library to connect to the database.
|
|
||||||
import mysql.connector
|
import mysql.connector
|
||||||
import json
|
import json
|
||||||
```
|
```
|
||||||
|
Afterward, a function would be called related to the "mysql.connector" library, where you would provide the login credentials of the database you have set up to establish a solid connection. This connection will be utilized multiple times later in the code.
|
||||||
Then we began the process of connecting with both the websocket and the database.
|
```py
|
||||||
|
|
||||||
First-off, we began making a connection to the database by using a mysql library in wich we gave the log in in order to connect to our ow database.
|
|
||||||
|
|
||||||
```js
|
|
||||||
//the data that has to be pushed needs to be
|
|
||||||
async def process_data(data):
|
async def process_data(data):
|
||||||
try:
|
try:
|
||||||
mydb = mysql.connector.connect(
|
mydb = mysql.connector.connect(
|
||||||
host="localhost",
|
host="localhost",
|
||||||
user="*****",
|
user="root",
|
||||||
password="*********",
|
# pasword hidden for privacy
|
||||||
database="*******"
|
password="********",
|
||||||
|
database="NodeData"
|
||||||
)
|
)
|
||||||
|
cursor = mydb.cursor()
|
||||||
|
```
|
||||||
|
This next part has a lot of different functions, so it will be split up for clarity.
|
||||||
|
|
||||||
|
It begins with creating a variable to retrieve information from a specific part of the database. This information is then stored in an array later on. In this case, it is selecting the existing MAC addresses from the database.
|
||||||
|
|
||||||
|
Afterward, a query is made for a different part of the code and acts as a "mold" for the data to be sent to the database. The values are not inserted yet because these will be the data collected from the nodes.
|
||||||
|
```py
|
||||||
|
#variable to connect to the DB
|
||||||
|
MACDataReading = mydb.cursor()
|
||||||
|
#get data from DB
|
||||||
|
MACDataReading.execute("SELECT MAC FROM Node")
|
||||||
|
#make a mold for the data to get to the DB
|
||||||
|
query = "INSERT INTO `Measurement` (NodeID, Type, Value) VALUES (%s, %s, %s)"
|
||||||
```
|
```
|
||||||
|
|
||||||
Then after this we code in the infromation we want to put inside of the database.
|
```py
|
||||||
The data collected from the websocket is json data, so this has to be changed.
|
processedData = json.loads(data)
|
||||||
|
processedTemp = (processedData['Temp'])
|
||||||
|
processedHumi = (processedData['Humi'])
|
||||||
|
processedeCO2 = (processedData['eCO2'])
|
||||||
|
processedTvoc = (processedData['TVOC'])
|
||||||
|
processedMAC = (processedData['node'])
|
||||||
|
MACTuple = (processedMAC,)
|
||||||
|
|
||||||
```js
|
MACDataFetching = MACDataReading.fetchall()
|
||||||
//Making a variable for the database connection
|
MACArray = list(MACDataFetching)
|
||||||
cursor = mydb.cursor()
|
|
||||||
//Making a variable for the database connection
|
|
||||||
MACDataReading = mydb.cursor()
|
|
||||||
//find the correct section and interact with it.
|
|
||||||
MACDataReading.execute("SELECT MAC FROM Node")
|
|
||||||
//the query for what needs to be inserted into the database ( atleast where it has to go).
|
|
||||||
query = "INSERT INTO `Measurement` (NodeID, Type, Value) VALUES (%s, %s, %s)"
|
|
||||||
//the recieved data from the websocket is json data so needs to be changed.
|
|
||||||
processedData = json.loads(data)
|
|
||||||
//variables about the recieved data points.
|
|
||||||
processedTemp = (processedData['Temp'])
|
|
||||||
processedHumi = (processedData['Humi'])
|
|
||||||
processedECo = (processedData['eCO2'])
|
|
||||||
processedTvoc = (processedData['TVOC'])
|
|
||||||
processedMAC = (processedData['node'])
|
|
||||||
//change the recieved macadress to a tuple.
|
|
||||||
MACTuple = (processedMAC,)
|
|
||||||
//fetch the data from the database an[d put it in an array.
|
|
||||||
MACDataFetching = MACDataReading.fetchall()
|
|
||||||
MACArray = list(MACDataFetching)
|
|
||||||
|
|
||||||
//see if the fetched data is not in the gotten array.
|
if MACTuple not in MACArray:
|
||||||
//otehrwise insert it into the database directly.
|
addingNode = "INSERT INTO `Node` (MAC) VALUES (%s)"
|
||||||
if MACTuple not in MACArray:
|
cursor.execute(addingNode, MACTuple)
|
||||||
addingNode = "INSERT INTO `Node` (MAC) VALUES (%s)"
|
mydb.commit()
|
||||||
cursor.execute(addingNode, MACTuple)
|
|
||||||
mydb.commit()
|
pushingDataArray = [(1, "Temp", processedTemp), (1, "Humi", processedHumi), (1, "eCO2", processedeCO2), (1, "TVOC", processedTvoc)]
|
||||||
//the websocket data that needs to be sent to the database.
|
for i in pushingDataArray:
|
||||||
pushingDataArray = [(1, "Temp", processedTemp), (1, "Humi", processedHumi), (1, "eCO2", processedECo), (1, "TVOC", processedTvoc)]
|
print(query ,i)
|
||||||
// forloop, go allong the array.
|
cursor.execute(query, i)
|
||||||
for i in pushingDataArray:
|
mydb.commit()
|
||||||
print(query ,i)
|
|
||||||
cursor.execute(query, i)
|
|
||||||
mydb.commit()
|
|
||||||
// show me the error it there is one.
|
|
||||||
except mysql.connector.Error as err:
|
except mysql.connector.Error as err:
|
||||||
print("MySQL Error:", err)
|
print("MySQL Error:", err)
|
||||||
finally:
|
finally:
|
||||||
cursor.close()
|
cursor.close()
|
||||||
mydb.close()
|
mydb.close()
|
||||||
```
|
|
||||||
|
|
||||||
After fully connecting t othe database, making statements of what to put there and telling the code what to do, we ofcourse need to write the code to connect to the weebsocket.
|
|
||||||
We begin by telling our websocket id and what type of port we are using.
|
|
||||||
Then we will collect live data from the conected websocket, store it in a variable, and then in the previous code
|
|
||||||
|
|
||||||
```js
|
|
||||||
//here the connection to the websocked is made
|
|
||||||
async def receive_data():
|
async def receive_data():
|
||||||
uri = "****************"
|
uri = "ws://145.92.8.114/ws"
|
||||||
|
|
||||||
try:
|
try:
|
||||||
async with websockets.connect(uri) as websocket:
|
async with websockets.connect(uri) as websocket:
|
||||||
while True:
|
while True:
|
||||||
// the data collected from the websocket is.
|
|
||||||
data = await websocket.recv()
|
data = await websocket.recv()
|
||||||
// data recieved: conformation.
|
|
||||||
print(f"Received data: {data}")
|
print(f"Received data: {data}")
|
||||||
await process_data(data)
|
await process_data(data)
|
||||||
// error sowing.
|
|
||||||
except websockets.ConnectionClosedError as e:
|
except websockets.ConnectionClosedError as e:
|
||||||
print("WebSocket connection closed:", e)
|
print("WebSocket connection closed:", e)
|
||||||
|
|
||||||
|
Reference in New Issue
Block a user