Pandas DataFrame四种写入方法效率对比
2024-07-02 05:21:38
该测试程序是将protocol变量转换为DataFrame类型为了便于使用numpy等数学工具
ImuEncode_pb2 是使用protocol生成的model， 具体使用方法可参考Protocol buffer 指南。
这是我使用的一些变量的声明
import time
import ImuEncode_pb2
import numpy as np
import pandas as pd
from pandas.core.frame import DataFrame
from ImuSignal import LegString
from ImuSignal import LegPositionString
from ImuSignal import SignalAttributeDict
from OnlineClassifier import OnlineClassifierImuSignalDataset = ImuEncode_pb2.ImuSignalDataset()
for i in range(50):imusignal =ImuSignalDataset.imusignal.add()for j in range(4):signal = imusignal.signal.add()signal.TimeStamp = 0signal.acc.AccX = 0signal.acc.AccY = 0signal.acc.AccZ = 0signal.angle.Roll = 0signal.angle.Pitch = 0signal.anglerate.Course = 0signal.quaternion.v = 0signal.quaternion.x = 0signal.quaternion.y = 0signal.quaternion.z = 0signal.leg = ImuEncode_pb2.Signal.RTsignal.legposition = ImuEncode_pb2.Signal.Footrate = 100 #sample rate
period = 0.5 #classify period
Leg = ['RT','LT']
Leg_Position = ['Foot','Thigh']
angle = ['Roll','Pitch','Course']
accel = ['AccX','AccY','AccZ']
特别注意下面代码段的加入，这是对于内存的声明，我是随便赋值不知道Python变量的声明原理后续完善
SignalFrame = DataFrame()
for leg in Leg:for leg_position in Leg_Position:for signal in accel + angle:SignalFrame[leg_position+"_"+signal+"_"+leg] = np.arange(999999999,999999999+int(period*rate),1)/1000000该方法不简单并且耗时应该是最长的，测试耗时为： 0.09793972969055176
SignalFrame = DataFrame()
for leg in Leg:for leg_position in Leg_Position:for signal in accel + angle:SignalFrame[leg_position+"_"+signal+"_"+leg] = np.arange(999999999,999999999+int(period*rate),1)/1000000starttime = time.time()
SignalFrame = pd.concat([pd.DataFrame(data={"Foot_AccX_RT":imusignal.signal[0].acc.AccX, "Foot_AccY_RT":imusignal.signal[0].acc.AccY, "Foot_AccZ_RT":imusignal.signal[0].acc.AccZ,\"Foot_Roll_RT":imusignal.signal[0].angle.Roll, "Foot_Pitch_RT":imusignal.signal[0].angle.Pitch, "Foot_Course_RT":imusignal.signal[0].anglerate.Course,\"Thigh_AccX_RT":imusignal.signal[1].acc.AccX, "Thigh_AccY_RT":imusignal.signal[1].acc.AccY, "Thigh_AccZ_RT":imusignal.signal[1].acc.AccZ,\"Thigh_Roll_RT":imusignal.signal[1].angle.Roll, "Thigh_Pitch_RT":imusignal.signal[1].angle.Pitch, "Thigh_Course_RT":imusignal.signal[1].anglerate.Course,\"Foot_AccX_LT":imusignal.signal[2].acc.AccX, "Foot_AccY_LT":imusignal.signal[2].acc.AccY, "Foot_AccZ_LT":imusignal.signal[2].acc.AccZ,\"Foot_Roll_LT":imusignal.signal[2].angle.Roll, "Foot_Pitch_LT":imusignal.signal[2].angle.Pitch, "Foot_Course_LT":imusignal.signal[2].anglerate.Course,\"Thigh_AccX_LT":imusignal.signal[3].acc.AccX, "Thigh_AccY_LT":imusignal.signal[3].acc.AccY, "Thigh_AccZ_LT":imusignal.signal[3].acc.AccZ,\"Thigh_Roll_LT":imusignal.signal[3].angle.Roll, "Thigh_Pitch_LT":imusignal.signal[3].angle.Pitch, "Thigh_Course_LT":imusignal.signal[3].anglerate.Course,\},index=[0]) for imusignal in ImuSignalDataset.imusignal],ignore_index=True)
endtime = time.time()
print("cost time:",endtime-starttime)
该方法不简单并且耗时长，测试耗时为： 0.04797029495239258
SignalFrame = DataFrame()for leg in Leg:for leg_position in Leg_Position:for signal in accel + angle:SignalFrame[leg_position+"_"+signal+"_"+leg] = np.arange(999999999,999999999+int(period*rate),1)/1000000starttime = time.time()
for imusignal in ImuSignalDataset.imusignal:SignalFrame = SignalFrame.append({"Foot_AccX_RT":imusignal.signal[0].acc.AccX, "Foot_AccY_RT":imusignal.signal[0].acc.AccY, "Foot_AccZ_RT":imusignal.signal[0].acc.AccZ,\"Foot_Roll_RT":imusignal.signal[0].angle.Roll, "Foot_Pitch_RT":imusignal.signal[0].angle.Pitch, "Foot_Course_RT":imusignal.signal[0].anglerate.Course,\"Thigh_AccX_RT":imusignal.signal[1].acc.AccX, "Thigh_AccY_RT":imusignal.signal[1].acc.AccY, "Thigh_AccZ_RT":imusignal.signal[1].acc.AccZ,\"Thigh_Roll_RT":imusignal.signal[1].angle.Roll, "Thigh_Pitch_RT":imusignal.signal[1].angle.Pitch, "Thigh_Course_RT":imusignal.signal[1].anglerate.Course,\"Foot_AccX_LT":imusignal.signal[2].acc.AccX, "Foot_AccY_LT":imusignal.signal[2].acc.AccY, "Foot_AccZ_LT":imusignal.signal[2].acc.AccZ,\"Foot_Roll_LT":imusignal.signal[2].angle.Roll, "Foot_Pitch_LT":imusignal.signal[2].angle.Pitch, "Foot_Course_LT":imusignal.signal[2].anglerate.Course,\"Thigh_AccX_LT":imusignal.signal[3].acc.AccX, "Thigh_AccY_LT":imusignal.signal[3].acc.AccY, "Thigh_AccZ_LT":imusignal.signal[3].acc.AccZ,\"Thigh_Roll_LT":imusignal.signal[3].angle.Roll, "Thigh_Pitch_LT":imusignal.signal[3].angle.Pitch, "Thigh_Course_LT":imusignal.signal[3].anglerate.Course,\},ignore_index=True)
endtime = time.time()
print("cost time:",endtime-starttime)
该方法简单并且耗时仅长于最快的实现方法，测试耗时为：0.016989469528198242
SignalFrame = DataFrame()for leg in Leg:for leg_position in Leg_Position:for signal in accel + angle:SignalFrame[leg_position+"_"+signal+"_"+leg] = np.arange(999999999,999999999+int(period*rate),1)/1000000
starttime = time.time()
i=0
for imusignal in ImuSignalDataset.imusignal:for SignalData in imusignal.signal:signal_dict = SignalAttributeDict(SignalData)for signal in angle + accel:SignalFrame.at[i,LegPositionString(SignalData.legposition)+"_"+signal+"_"+LegString(SignalData.leg)]=signal_dict[signal]i+=1
endtime = time.time()
print("cost time:",endtime-starttime)
该方法不简单，如果数据量较大可能会影响数据的准确度，但是耗时应该是最短的，测试耗时为：0.012990951538085938
SignalFrame = DataFrame()for leg in Leg:for leg_position in Leg_Position:for signal in accel + angle:SignalFrame[leg_position+"_"+signal+"_"+leg] = np.arange(999999999,999999999+int(period*rate),1)/1000000starttime = time.time()
i=0
for imusignal in ImuSignalDataset.imusignal:SignalFrame.loc[i]=[imusignal.signal[0].acc.AccX , imusignal.signal[0].acc.AccY , imusignal.signal[0].acc.AccZ,\imusignal.signal[0].angle.Roll , imusignal.signal[0].angle.Pitch , imusignal.signal[0].anglerate.Course,\imusignal.signal[1].acc.AccX , imusignal.signal[1].acc.AccY , imusignal.signal[1].acc.AccZ,\imusignal.signal[1].angle.Roll , imusignal.signal[1].angle.Pitch , imusignal.signal[1].anglerate.Course,\imusignal.signal[2].acc.AccX , imusignal.signal[2].acc.AccY , imusignal.signal[2].acc.AccZ,\imusignal.signal[2].angle.Roll , imusignal.signal[2].angle.Pitch , imusignal.signal[2].anglerate.Course,\imusignal.signal[3].acc.AccX , imusignal.signal[3].acc.AccY , imusignal.signal[3].acc.AccZ,\imusignal.signal[3].angle.Roll , imusignal.signal[3].angle.Pitch , imusignal.signal[3].anglerate.Course]i+=1
endtime = time.time()
print("cost time:",endtime-starttime)
用于生成ImuEncode_pb2的ImuEncode.proto文件如下
syntax = "proto2";
package ImuTutorial;message Signal {enum Leg {RT = 0;LT = 1;}enum LegPosition{Foot = 0;Thigh = 1;}required Leg leg = 1 [default = RT];required LegPosition legposition = 2 [default = Foot];optional float TimeStamp = 3;optional int64 time = 4;optional int32 id = 5;message Acc{required float AccX = 1;required float AccY = 2;required float AccZ = 3;}message Angle{optional float Course = 1;required float Roll = 2;required float Pitch = 3;}message AngleRate{required float Course = 1;optional float Roll = 2;optional float Pitch = 3;}message Quaternion{optional float v = 1;required float x = 2;required float y = 3;required float z = 4;}required Acc acc = 6;required Angle angle= 7;required AngleRate anglerate= 8;optional Quaternion quaternion = 9;
}message ImuSignal {required uint64 time = 1;repeated Signal signal = 2;
}message ImuSignalDataset {repeated ImuSignal imusignal = 1;
}message Datalength {required fixed32 length = 1;
}
Pandas DataFrame四种写入方法效率对比相关推荐
- python最快的循环方法_【转】【Python效率】五种Pandas循环方法效率对比
[Python效率]五种Pandas循环方法效率对比 - 文兄的文章 - 知乎 https://zhuanlan.zhihu.com/p/80880493 正文: 如果你使用过Python及Panda ...
- 循环下标_【转】【Python效率】五种Pandas循环方法效率对比
[Python效率]五种Pandas循环方法效率对比 - 文兄的文章 - 知乎 https://zhuanlan.zhihu.com/p/80880493 正文: 如果你使用过Python及Panda ...
- mybatis批量更新数据三种方法效率对比
探讨批量更新数据三种写法的效率问题. 实现方式有三种, 1> 用for循环通过循环传过来的参数集合,循环出N条sql,需要在db链接url后面带一个参数 &allowMultiQuer ...
- python 财务分析可视化方法_Python数据可视化的四种简易方法
Python数据可视化的四种简易方法 作者:PHPYuan 时间:2018-11-28 03:40:43 摘要: 本文讲述了热图.二维密度图.蜘蛛图.树形图这四种Python数据可视化方法. 数据可视 ...
- python下载文件到本地-Python下载网络文本数据到本地内存的四种实现方法示例
本文实例讲述了Python下载网络文本数据到本地内存的四种实现方法.分享给大家供大家参考,具体如下: import urllib.request import requests from io imp ...
- 大数据可视化python_大数据分析之Python数据可视化的四种简易方法
本篇文章探讨了大数据分析之Python数据可视化的四种简易方法,希望阅读本篇文章以后大家有所收获,帮助大家对相关内容的理解更加深入. < 数据可视化是任何数据科学或机器学习项目的一个重要组成部分 ...
- EF、Chloe、SqlSugar、DOS.ORM四种ORM框架的对比
文章目录 四种orm框架的对比 1.什么是ORM? 2.四种框架的基本信息对比如下表: 3.四种框架的耗时情况如下: 4.四种框架都支持哪些功能? 5.就个人使用而言,比较推荐哪个框架? 6.SqlS ...
- 计算机无法连接无线信号,win7系统连接无线信号时提示Windows无法连接到路由器名称的四种解决方法...
现如今网络发展速度非常快,无线网络已经普及了,使用率高了遇到的问题也就多了.比如有时候笔记本win7系统连接无线信号时出现"Windows无法连接到路由器名称"(如下图所示),该如 ...
- 电脑快捷方式删除文件后四种找回方法
快捷指令是一种用作替代快捷键操作的技术.也可以称为"快捷键","快捷方式"或"快捷键序列",它们允许用户在非常快速和方便的方式建立特定操作序 ...
最新文章
- 开发日记-20190822 关键词 读书笔记《Unix环境高级编程（第二版）》《掌控习惯》DAY 2
- python快速入门答案-总算懂得python脚本快速入门教程
- 有向图的深度和广度遍历
- Linux 查看网卡流量【转】
- linux was8.5 配db2数据源配置,DataStage v8.5对于db2 v8数据库的支持问题
- C 库函数 - pow()
- Linux桌面虚拟化技术KVM
- Ubuntu安装完驱动后进不去界面
- Abaqus简单部件受力分析
- HC-05嵌入式蓝牙串口通讯
- python饼状图显示其比例_Python学习笔记(matplotlib篇)--使用matplotlib绘制饼状图
- Android仿网易云项目代码,Android项目实战教程之高仿网易云音乐启动页实例代码...
- Matlab中的ttest2()函数实现
- java作为微信小程序的后端_微信小程序连接java后端
- js两只手指控制div图片放大缩小功能（2）
- equals和hashcode方法
- SQL Server数据库实操 第四波 视图、索引
- 小程序生成海报并下载
- Ubuntu18.04 安装 omnet 5.6.2
- Java开发规范整理