自拍偷在线精品自拍偷,亚洲欧美中文日韩v在线观看不卡

基于PHP擴展Phpy實現(xiàn)Apache ORC高效讀寫，搞定大數(shù)據(jù)處理

作者：Tinywan
大數(shù)據(jù) Hadoop
Apache ORC 是一種專為 Hadoop 生態(tài)系統(tǒng)設(shè)計的列式存儲文件格式，它具備高效的數(shù)據(jù)壓縮和快速的查詢性能。由于采用列式存儲，ORC 格式在處理大規(guī)模數(shù)據(jù)時能夠顯著減少 I/O 開銷，提高數(shù)據(jù)處理效率，廣泛應(yīng)用于大數(shù)據(jù)處理框架如 Apache Hive、Apache Spark 等。

概述

用于讀取和寫入 Apache ORC 文件格式的 PHP 庫。它使用 swoole/phpy 擴展來調(diào)用 Python 模塊 PyORC 以實現(xiàn)相關(guān)功能。

Apache ORC

Apache ORC 是一種專為 Hadoop 生態(tài)系統(tǒng)設(shè)計的列式存儲文件格式，它具備高效的數(shù)據(jù)壓縮和快速的查詢性能。由于采用列式存儲，ORC 格式在處理大規(guī)模數(shù)據(jù)時能夠顯著減少 I/O 開銷，提高數(shù)據(jù)處理效率，廣泛應(yīng)用于大數(shù)據(jù)處理框架如 Apache Hive、Apache Spark 等。

特性：

  • 讀取 ORC 文件：PyORC 允許 Python 程序從 ORC 文件中讀取數(shù)據(jù)。它支持按行或按列讀取數(shù)據(jù)，并且可以根據(jù)需要選擇讀取特定的列，從而避免不必要的數(shù)據(jù)加載，提升讀取效率。
  • 寫入 ORC 文件：能夠?qū)?Python 中的數(shù)據(jù)（如列表、字典等）以 ORC 格式寫入文件。在寫入過程中，可以指定數(shù)據(jù)的模式（schema），包括列名、數(shù)據(jù)類型等，以確保數(shù)據(jù)的正確存儲。
  • 數(shù)據(jù)模式處理：支持對 ORC 文件的數(shù)據(jù)模式進行解析和操作。可以獲取文件的模式信息，也可以自定義模式來創(chuàng)建新的 ORC 文件。

Swoole Phpy

phpy 是識沃團隊最新推出的開源項目，目標是為 PHP 引入 Python 生態(tài)，來彌補 PHP 生態(tài)的空缺和不足。phpy 使得 PHP 可以調(diào)用所有 Python 的包。

它為 PHP 開發(fā)者提供了一種在 PHP 代碼里調(diào)用 Python 模塊和函數(shù)的能力，搭建起了 PHP 和 Python 這兩種編程語言之間的橋梁，極大地擴展了 PHP 的功能邊界。

特性：

  1. 無縫調(diào)用 Python 代碼：借助 phpy 擴展，PHP 開發(fā)者能夠在 PHP 腳本里直接調(diào)用 Python 編寫的函數(shù)、類和模塊。這意味著可以利用 Python 豐富的科學計算、機器學習、數(shù)據(jù)分析等領(lǐng)域的庫，比如 NumPy、SciPy、TensorFlow 等，而無需離開 PHP 開發(fā)環(huán)境。
  2. 數(shù)據(jù)類型映射：phpy 實現(xiàn)了 PHP 和 Python 數(shù)據(jù)類型之間的自動映射。例如，PHP 的數(shù)組可以自動轉(zhuǎn)換為 Python 的列表或字典，PHP 的標量類型（如整數(shù)、浮點數(shù)、字符串等）也能正確地轉(zhuǎn)換為 Python 對應(yīng)的類型，反之亦然。這使得在兩種語言之間傳遞數(shù)據(jù)變得非常方便。
  3. 高性能：phpy 在設(shè)計上注重性能優(yōu)化，通過高效的底層實現(xiàn)，減少了 PHP 和 Python 之間的交互開銷，確保數(shù)據(jù)傳遞和函數(shù)調(diào)用的高效性。

安裝

Composer安裝

composer require workbunny/php-orc

查看幫助

./vendor/bin/php-orc --help
Description:
List commands

Usage:
list [options] [--] [<namespace>]

Arguments:
namespace             Thenamespacename

Options:
      --raw             Tooutputrawcommandlist
      --format=FORMAT   Theoutputformat (txt, xml, json, ormd) [default: "txt"]
      --short           Toskipdescribingcommands' arguments
  -h, --help            Displayhelpforthegivencommand. Whennocommandisgivendisplayhelpforthelistcommand
      --silent          Donotoutputanymessage
  -q, --quiet           Onlyerrorsaredisplayed. Allotheroutputissuppressed
  -V, --version         Displaythisapplicationversion
      --ansi|--no-ansiForce (ordisable --no-ansi) ANSIoutput
  -n, --no-interactionDonotaskanyinteractivequestion
  -v|vv|vvv, --verboseIncreasetheverbosityofmessages: 1 fornormaloutput, 2 formoreverboseoutputand 3 fordebug

依賴環(huán)境安裝

安裝Python

通過.vendor/bin/php-orc install:python安裝

./vendor/bin/php-orc install:python
[>] Checking and installing Python 3.10+ ...
[i] Python 3.10+ is already installed.
[√] Python installation complete.

安裝成功后，查看安裝成功的python版本號

# python -V
Python 3.12.8

安裝PHPy拓展

通過.vendor/bin/php-orc install:phpy安裝

./vendor/bin/php-orc install:phpy
[?] Please specify the installation directory (default: .runtime): 

[>] Downloading the latest source code ...
Cloning into '/var/www/webman/workerman5.1/.runtime/swoole_phpy_latest'...
remote: Enumerating objects: 827, done.
remote: Counting objects: 100% (827/827), done.
remote: Compressing objects: 100% (516/516), done.
remote: Total 827 (delta 327), reused 642 (delta 289), pack-reused 0 (from 0)
Receiving objects: 100% (827/827), 560.36 KiB | 1.74 MiB/s, done.
Resolving deltas: 100% (327/327), done.
Updating files: 100% (749/749), done.
warning: the following paths have collided (e.g. case-sensitive paths
on a case-insensitive filesystem) and only one from the same
colliding group is in the working tree:
'lib/python/ast/BoolOp.php'
'lib/python/ast/boolop.php'
'lib/python/ast/ExceptHandler.php'
'lib/python/ast/excepthandler.php'
'lib/python/ast/Expr.php'
'lib/python/ast/expr.php'
'lib/python/ast/Mod.php'
'lib/python/ast/mod.php'
'lib/python/ast/Slice.php'
'lib/python/ast/slice.php'
'lib/python/ast/UnaryOp.php'
'lib/python/ast/unaryop.php'
[>] Installing dependencies...
[>] fetch https://mirrors.ustc.edu.cn/alpine/v3.21/main/x86_64/APKINDEX.tar.gz
[>] fetch https://mirrors.ustc.edu.cn/alpine/v3.21/community/x86_64/APKINDEX.tar.gz
[>] OK: 422 MiB in 104 packages
...
[√] PHPy installation completed successfully.

安裝成功后，查看安裝成功的phpy擴展和版本號

# php -m |grep phpy
phpy

查看phpy版本號

# php --ri phpy

phpy

phpy support => enabled
Copyright => 上海識沃網(wǎng)絡(luò)科技有限公司
Email => service@swoole.com
Website => https://www.swoole.com/
Extension Version => 1.0.10
Python Version => 3.12.8 (main, Dec  7 2024, 05:56:13) [GCC 14.2.0]

安裝pyorc模塊

通過.vendor/bin/php-orc install:pyorc安裝

./vendor/bin/php-orc install:pyorc
[>] Checking and installing Python PyORC ...
[>] Upgrading pip ...
[i] pip is already up-to-date.
[>] Installing TZData ...
WARNING: Package(s) not found: tzdata
[>] Collecting tzdata
[>] Downloading tzdata-2025.1-py2.py3-none-any.whl.metadata (1.4 kB)
[>] Downloading tzdata-2025.1-py2.py3-none-any.whl (346 kB)
[>] Installing collected packages: tzdata
[>] Successfully installed tzdata-2025.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
[>] Installing PyORC-latest ...
WARNING: Package(s) not found: pyorc
[>] Collecting pyorc
[>] Downloading pyorc-0.10.0-cp312-cp312-musllinux_1_2_x86_64.whl.metadata (3.0 kB)
[>] Downloading pyorc-0.10.0-cp312-cp312-musllinux_1_2_x86_64.whl (3.8 MB)
[>] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.8/3.8 MB 8.1 MB/s eta 0:00:00
[>] Installing collected packages: pyorc
[>] Successfully installed pyorc-0.10.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
[√] Python and PyORC installation complete.

開源技術(shù)小棧

安裝過程如果提示以下錯誤：

[>] Checking and installing Python PyORC ...
[>] Upgrading pip ...
sh: pip: not found

請執(zhí)行以下命令，安裝 pip

apk update
apk add py3-pip

執(zhí)行案例

PHPY案例

os.php文件

<?php
/**
 * @desc 打印操作系統(tǒng)信息
 * @author Tinywan(ShaoBo Wan)
 */
declare(strict_types=1);

function main(): void
{
    $m = PyCore::import("os");
    var_dump($m instanceof PyObject);
    $rs = $m->uname();
    echo $rs . PHP_EOL;
    echo $rs->version . PHP_EOL;
}

main();

執(zhí)行輸出

# php os.php 
bool(true)
posix.uname_result(sysname='Linux', nodename='b6fb15fcfed9', release='5.10.102.1-microsoft-standard-WSL2', version='#1 SMP Wed Mar 2 00:30:59 UTC 2022', machine='x86_64')
#1 SMP Wed Mar 2 00:30:59 UTC 2022

PyORC案例

文件讀取 reader.php

<?php
/**
 * @desc 文件讀取
 * @author Tinywan(ShaoBo Wan)
 * @date 2025/2/26 22:51
 */
declare(strict_types=1);

usefunctionWorkbunny\PhpOrc\open;

require_once__DIR__ . '/vendor/autoload.php';

$reader = new \Workbunny\PhpOrc\Reader(open(__DIR__ . '/vendor/workbunny/php-orc/examples/example-php.orc','rb'));
// header schema
var_dump(
// 總行數(shù)
    $reader->count(),
    // 頭格式
    $reader->schema(),
    // 壓縮類型
    $reader->compression(),
    // 元數(shù)據(jù)
    $reader->userMetadata(),
    // 寫入方式
    $reader->writerId(),
    // 寫入版本類型
    $reader->writerVersion(),
    // 基座版本
    $reader->softwareVersion(),
    // 格式版本
    $reader->formatVersion()
);
// 迭代
$reader->iteration(function ($i, $row) {
    var_dump($i, $row);
});

執(zhí)行輸出

# php reader.php 
int(3)
string(52) "struct<id:int,group:string,name:string,email:string>"
int(0)
array(3) {
'group' =>
  string(9) "workbunny"
'user' =>
  string(9) "chaz6chez"
'email' =>
  string(25) "chaz6chez1993@outlook.com"
}
string(14) "ORC_CPP_WRITER"
int(6)
string(13) "ORC C++ 2.1.0"
array(2) {
  [0] =>
  int(0)
  [1] =>
  int(12)
}
int(0)
array(4) {
  [0] =>
  int(1)
  [1] =>
  string(9) "workbunny"
  [2] =>
  string(5) "test1"
  [3] =>
  string(19) "test1@workbunny.com"
}
int(1)
array(4) {
  [0] =>
  int(2)
  [1] =>
  string(9) "workbunny"
  [2] =>
  string(5) "test2"
  [3] =>
  string(19) "test2@workbunny.com"
}
int(2)
array(4) {
  [0] =>
  int(3)
  [1] =>
  string(9) "workbunny"
  [2] =>
  string(5) "test3"
  [3] =>
  string(19) "test3@workbunny.com"
}

責任編輯：武曉燕 來源： 開源技術(shù)小棧
ApacheORCHadoop
相關(guān)推薦

2025-02-05 08:13:48

Go語言范式

2018-12-07 14:50:35

大數(shù)據(jù)數(shù)據(jù)采集數(shù)據(jù)庫

2020-11-02 15:56:04

大數(shù)據(jù)數(shù)據(jù)庫技術(shù)

2023-08-25 15:13:16

大數(shù)據(jù)云計算

2023-07-12 12:02:06

WOT大數(shù)據(jù)流式數(shù)據(jù)湖

2016-11-14 18:17:42

Apache Beam大數(shù)據(jù)

2021-04-02 12:55:14

數(shù)據(jù)處理Apache Falc開發(fā)

2021-07-20 15:37:37

數(shù)據(jù)開發(fā)大數(shù)據(jù)Spark

2017-07-21 14:22:17

大數(shù)據(jù)大數(shù)據(jù)平臺數(shù)據(jù)處理

2020-07-22 08:13:22

大數(shù)據(jù)

2022-11-17 11:52:35

pandasPySpark大數(shù)據(jù)

2012-11-28 20:56:03

浪潮一體機浪潮大數(shù)據(jù)

2023-10-05 12:43:48

數(shù)據(jù)處理

2023-11-29 13:56:00

數(shù)據(jù)技巧

2011-09-01 15:12:43

SQL ServerHadoop

2015-05-05 11:18:18

大數(shù)據(jù)Hadoop技術(shù)處理

2015-12-07 15:05:39

UCloudUHadoop大數(shù)據(jù)

2016-02-17 09:55:25

SMACK數(shù)據(jù)處理可擴展架構(gòu)

2015-10-16 09:50:10

2015-11-09 09:58:31

大數(shù)據(jù)Lambda架構(gòu)
點贊
收藏

51CTO技術(shù)棧公眾號