文件上传:利用pandas读取和django_celery异步上传到服务器上的文件


发布时间:2020-01-10 08:20    作者: Uncle Hui   已过去:10 月,3 周   阅读总量:424 已被赞:0


结合上篇文章,我们来实现上传文件的读取:

http://www.hui2017.top/blog/piyyqo7WHBkNinyFR3cXKc/

首先安装两个包:

pip install pandas

(dj_evn) root@instance-1xl5vd0y:~# pip install pandas
Looking in indexes: https://mirrors.aliyun.com/pypi/simple/
Collecting pandas
  Downloading https://mirrors.aliyun.com/pypi/packages/52/3f/f6a428599e0d4497e1595030965b5ba455fd8ade6e977e3c819973c4b41d/pandas-0.25.3-cp36-cp36m-manylinux1_x86_64.whl (10.4MB)
     |████████████████████████████████| 10.4MB 7.4MB/s 
Collecting python-dateutil>=2.6.1
  Downloading https://mirrors.aliyun.com/pypi/packages/d4/70/d60450c3dd48ef87586924207ae8907090de0b306af2bce5d134d78615cb/python_dateutil-2.8.1-py2.py3-none-any.whl (227kB)
     |████████████████████████████████| 235kB 42.9MB/s 
Collecting numpy>=1.13.3
  Downloading https://mirrors.aliyun.com/pypi/packages/62/20/4d43e141b5bc426ba38274933ef8e76e85c7adea2c321ecf9ebf7421cedf/numpy-1.18.1-cp36-cp36m-manylinux1_x86_64.whl (20.1MB)
     |████████████████████████████████| 20.2MB 381kB/s 
Requirement already satisfied: pytz>=2017.2 in ./.virtualenvs/dj_evn/lib/python3.6/site-packages (from pandas) (2019.3)
Collecting six>=1.5
  Downloading https://mirrors.aliyun.com/pypi/packages/65/26/32b8464df2a97e6dd1b656ed26b2c194606c16fe163c695a992b36c11cdf/six-1.13.0-py2.py3-none-any.whl
Installing collected packages: six, python-dateutil, numpy, pandas
Successfully installed numpy-1.18.1 pandas-0.25.3 python-dateutil-2.8.1 six-1.13.0
 

 pip install xlrd

(dj_evn) root@instance-1xl5vd0y:~# pip install xlrd
Looking in indexes: https://mirrors.aliyun.com/pypi/simple/
Collecting xlrd
  Downloading https://mirrors.aliyun.com/pypi/packages/b0/16/63576a1a001752e34bf8ea62e367997530dc553b689356b9879339cf45a4/xlrd-1.2.0-py2.py3-none-any.whl (103kB)
     |████████████████████████████████| 112kB 1.7MB/s 
Installing collected packages: xlrd
Successfully installed xlrd-1.2.0
 

 views.py

导入:

import pandas as pd

这样读取:

df = pd.read_excel("{}/media/my_file/{}".format(BASE_DIR,file.name))
print(df)
print(df.columns) #打印表格文件的列

安装异步所需要的插件:

pip install django-celery

pip install redis==2.10.6

注册:

INSTALLED_APPS = [

'djcelery',

]

settings.py 配置异步:

#配置异步///////////////////////开始
import djcelery
djcelery.setup_loader()
#BROKER_URL = 'amqp://guest:guest@localhost:5672//'
#CELERY_RESULT_BACKEND = 'redis://localhost:6379/1'
BROKER_URL = 'redis://localhost:6379/1'
CELERY_RESULT_BACKEND = 'redis://localhost:6379/2'
from kombu import Queue
CELERY_DEFAULT_QUEUE = 'worker_queue'
CELERY_QUEUE =(
    Queue('worker_queue',routing_key='worker_queue')
)
#配置异步///////////////////////结束

在app里新建一个python文件必须取名叫tasks.py

异步任务都写在里面,会自动来这里寻找执行

tasks.py:

from .models import New_category,News
from celery import task #引入异步

 

@task
def da_new(df):
    df['author_id'] = df.apply(lambda x:sr_new(x['category']),axis=1)  #将df(导入表)里的数据放入,并通过分类名找到ID并存进去,category为导入表的分类标题
    df = df[['name', 'neirong', 'c_time', 'author_id']] #将导入表里需要的列写进来,author_id为上面查询出的
  #  df = df.rename(columns={'c_time': 'new_time'}) #如果导入列与数据库表的名字不同,可以使用这个方法改名
    for idx, row in df.iterrows():  #遍历出数据
        name = row['name'] #数据在row里,分别获取出来
        neirong = row['neirong']
        c_time = row['c_time']
        categroy_id = row['author_id']
        categroy_id = str(categroy_id)
        if categroy_id != 'nan':  #sr_new这个函数里,将没有的ID传了None,所以这里要判断
            categroy_id = int(float(categroy_id))
            cate = New_category.objects.get(id=categroy_id)
            new = News(name=name, neirong=neirong, c_time=c_time, author=cate)
            new.save()
        else:
            new = News(name=name, neirong=neirong, c_time=c_time)
            new.save()
    return 'success'
#将导入表里的分类名字放入下面的函数查询出ID并返回
def sr_new(sr):
    name = sr
    new_category=New_category.objects.filter(name=name).first() #分类表里查
    if new_category:
        pk =new_category.pk
    else:
        pk = None
    return pk

views.py 修改一下:

 

def piliang_add(request):
    if request.method == 'POST':
        file = request.FILES.get('exc_file')
        if file != None :
            f=open(os.path.join(BASE_DIR,'media','my_file',file.name),'wb')
            for chunk in file.chunks():
                f.write(chunk)
            f.close()
            df = pd.read_excel("{}/media/my_file/{}".format(BASE_DIR,file.name))
           #使用异步写入数据库(1-2)
            from .tasks import da_new #1
            da_new.delay(df)          #2
            return restful.ok()
        else:
           return restful.params_error(message='请选择excl文件上传!')
    else:
        return render(request, 'piliang_add.html')

 在环境里启动异步:

去环境里启动celery

(dj_evn) root@instance-1xl5vd0y:/home/dj_celery# python manage.py celery worker -l INFO

python manage.py celery worker -l INFO

 

Running a worker with superuser privileges when the

worker accepts messages serialized with pickle is a very bad idea!

If you really want to continue then you have to set the C_FORCE_ROOT

environment variable (but please think about this before you do).

User information: uid=0 euid=0 gid=0 egid=0

报上面这个错误,是说我们的服务器是使用的 root用户,所以最好换成非root用户来操作以上过程,当然我们也可以这样做:

在settings.py配置异步任务的代码里加入这一句

from celery import platforms
platforms.C_FORCE_ROOT = True

配置完成,上传文件测试!

 

点赞

0




登陆后方可评论