diff --git a/.codebuddy/memory/2026-06-01.md b/.codebuddy/memory/2026-06-01.md index e53bf9e..f9c744c 100644 --- a/.codebuddy/memory/2026-06-01.md +++ b/.codebuddy/memory/2026-06-01.md @@ -27,3 +27,27 @@ - prefetch 流程补充游标分页、参数构建、响应解析的详细说明 - 新增「并发保护」章节，记录内存锁和调度器时序 - 调度器章节补充 for+sleep 模式说明 + +## 钉钉平台新增 +- 创建 `seed_data_dingtalk.sql`：钉钉开放平台 + 部门列表接口 +- auth_type: OAUTH2，access_token 在 URL 查询参数中 +- 部门列表：POST + parameters_location=query + recursive 递归遍历，响应 `result` 为数组 +- 表名：`dingtalk_department`，冲突键：dept_id + +## 递归遍历支持 +- 新增 `RecursiveConfig` + `parseRecursiveConfig` + `syncRecursive` 函数 +- `syncRecursive`：BFS 队列遍历，先查根级 → 对每个子部门递归查下级，防重复 + 最大深度限制 +- `buildReqBody` / `buildPrefetchParams` 增加过滤 `recursive` / `max_recursive_depth` + +## 钉钉智能人事平台 +- 新建 `seed_data_dingtalk_hrm.sql`：钉钉智能人事专用平台 +- auth_type: OAUTH2 + Header 认证（`x-acs-dingtalk-access-token`），独立于 oapi 平台 +- 企业职位列表 `position_list`：POST + cursor 分页 + hasMore，API base `api.dingtalk.com` + +## 钉钉角色列表 + 引擎增强 +- `parseRespExt` 增加 `has_more_field` 支持 + 数字游标(float64→string) +- `syncWithPrefetch` 支持递归预取来源（BFS） +- `buildReqBody` 增加 `pagination_mode: "offset"` 支持（计算 offset = (page-1)*pageSize） +- `syncSingleAPI` 增加 hasMore 驱动的分页循环（钉钉 offset/size + hasMore） +- 新增 `hasMoreCheck` 辅助函数 +- 新增 `seed_data_dingtalk.sql` 角色列表接口：偏移分页 + hasMore diff --git a/.codebuddy/memory/2026-06-03.md b/.codebuddy/memory/2026-06-03.md new file mode 100644 index 0000000..ca4e28c --- /dev/null +++ b/.codebuddy/memory/2026-06-03.md @@ -0,0 +1,36 @@ +# 2026-06-03 工作记录 + +## 钉钉智能薪酬平台对接 +- 新增 `dingtalk_salary` 平台 + 2个接口到 `sql/seed_data_dingtalk_salary.sql` +- 新认证类型 `APP_SIGNATURE`：app-id + signature 头部认证 +- 签名算法：MD5(body_string + app_secret).toUpperCase() + +### 接口明细 +1. **salary_dept_list** — `POST /oapi/salary/dept/list`，无入参，单次拉取全量部门 +2. **salary_statistics** — `POST /oapi/salary/statistics/dept`，prefetch 自 salary_dept_list，row_inject statisticsMonth + +### 代码改动 +- `api_client.go`: `applyAuthHeader` 扩展支持 `APP_SIGNATURE` 认证，新增 `applyAppSignatureAuth` + `computeBodySignature` +- `platform_manager.go`: 新增 `APP_SIGNATURE` case，从 auth_config 提取 app_id/app_secret +- `dynamic_sync.go`: + - `toFloat64` 扩展支持 string→float64（适配 salary API 返回 code: "200" 字符串） + - `buildReqBody`/`buildPrefetchParams` skip list 新增 `row_inject` + - 新增 `injectRowFields` 函数：将请求参数注入响应行 + - `injectRowFields` 调用于 syncWithPrefetch 中3处 row 注入点 + +## 新增钉钉智能薪酬人力成本报表接口 (2026-06-03 第二阶段) + +### 新增接口 +3. **salary_statistic_report** — `POST /oapi/salary/statistic/report/data`，无分页，single_record，获取指定月份的人力成本报表 + - 入参：`calBizId`（格式 yyyyMM"M"，如 "202606M"，为空默认当月） + - 出参：`data.calBizId` + `data.sumStatisticsData`（成本项数组 JSONB） + - 表：`dingtalk_salary_statistic_report`，conflict_key: `calBizId` + +4. **salary_statistic_report_group** — `POST /oapi/salary/statistic/report/groupData`，无分页，single_record + - 入参：`calBizId` + `salaryGroupName`（必填），使用 `row_inject` 注入 salaryGroupName + - 表：`dingtalk_salary_statistic_report_group`，conflict_keys: `calBizId` + `salaryGroupName` + +### 代码改动 +- `dynamic_sync.go`：修复主同步路径（非 prefetch）中 `injectRowFields` 未调用的问题 + - 在 5 处 savePage 调用前添加 injectRowFields：初始页、游标分页循环、hasMore 分页、普通分页（有/无 response_config） + - 使 `row_inject` 在单记录（single_record）非 prefetch 接口中也能生效 diff --git a/.codebuddy/memory/2026-06-08.md b/.codebuddy/memory/2026-06-08.md new file mode 100644 index 0000000..af1c589 --- /dev/null +++ b/.codebuddy/memory/2026-06-08.md @@ -0,0 +1,15 @@ +# 2026-06-08 工作日志 + +## goview-report 项目初始化 +- 基于 GoView (gitee.com/bufanyun/go-view-server) 初始化报表引擎项目 +- 位置: `/Users/xujiaqian/GolandProjects/HDWL/goview-report/` +- 改造内容: + - 模块名 hotgo → goview-report + - MySQL 驱动替换为 PostgreSQL 驱动 (github.com/gogf/gf/contrib/drivers/pgsql/v2) + - Go 版本升级到 1.26.0，GoFrame 升级到 v2.10.0 + - config.yaml 中数据库连接改为 pgsql://postgres:root@127.0.0.1:5432/engine + - 端口保持 8090，与 data-engine(:3002) 不冲突 + - 元数据表前缀 hg_，data-engine 同步表无前缀，同库不同前缀 + - 创建 Dockerfile + k8s/deployment+service+configmap + - 编译验证通过 +- 与 data-engine 关系: 读写分离，goview-report 只读 PG 中的同步表做报表展示 diff --git a/.codebuddy/memory/2026-06-09.md b/.codebuddy/memory/2026-06-09.md new file mode 100644 index 0000000..297a004 --- /dev/null +++ b/.codebuddy/memory/2026-06-09.md @@ -0,0 +1,18 @@ +# 2026-06-09 工作日志 + +## 公共查询接口开发 +- 新增 `/public/query` POST 接口，支持： + - 字段白名单验证（只允许表定义中声明的字段） + - 表名白名单验证（只允许 api_interface 中有 table_definition 的表） + - WHERE 条件（支持 `_eq/_ne/_gt/_lt/_ge/_le/_like/_in/_between` 操作符） + - GROUP BY / ORDER BY / 分页 + - 强制 tenant_id = 1 租户隔离 +- 新增辅助接口： + - `GET /public/tables` - 获取可查询表列表 + - `GET /public/tables/{table}/columns` - 获取表字段列表 + - `DELETE /public/cache/clear` - 清除表缓存 +- 文件位置： + - `model/dto/public/public_query_dto.go` + - `service/public/public_query_service.go` + - `controller/public/public_query_controller.go` +- 同步修复 `dynamic_sync.go` 中残留的无效代码 "了呢" diff --git a/.codebuddy/memory/2026-06-10.md b/.codebuddy/memory/2026-06-10.md new file mode 100644 index 0000000..4d91aae --- /dev/null +++ b/.codebuddy/memory/2026-06-10.md @@ -0,0 +1,179 @@ +# 2026-06-10 + +## 通用报表引擎 - 补充前端 CRUD 能力 + +为 `common/report` 包补充了配置的 CRUD 接口，使前端可以直接维护报表配置，无需操作数据库。 + +### 新增文件变更 +- `model/model.go`: 新增 SaveBusinessReq/SaveReportReq/SaveFieldReq/SaveExtractConfigReq 等请求结构体，SaveResult/DeleteResult 响应结构体 +- `config/loader.go`: 新增 Create/Update/Delete + GetByID 方法（Business/Report/Field/ExtractConfig 四种配置） +- `api.go`: 新增 SaveBusiness/SaveReport/SaveField/SaveExtractConfig + Delete/Get 接口（新增/修改合一，id 为空则新增） +- `example_usage.go`: 新增第五、六节，CRUD 调用示例 + 任意新平台零代码接入流程 + +### 关键设计决策 +- **Save 合一**：通过 `ID *int64` 区分新增（nil）和修改（有值），简化前端调用 +- **软删除**：status → INACTIVE + deleted_at=NOW()，数据不丢失 +- **缓存失效**：写操作后自动失效对应业务/报表缓存，保证下次读取一致性 +- 所有 CRUD 方法先调用 initTables() 确保表存在 + +### example_usage.go 完整重写 +将示例文件从分节的注释风格重写为6个完整场景的 Go 函数示例： +1. **快手电商完整接入**（最详细，从业务注册到前端查询6步全流程） +2. **已有配置的 CRUD 二次开发参考**（业务/报表/字段/抽取 增删改查） +3. **任意平台接入通用模式**（淘宝示例，零硬编码） +4. **外部业务服务调用**（goview-report 等直接 import） +5. **Direct 模式示例**（不聚合的逐行抽取） +6. **前端 HTTP API 交互流程**（伪代码，展示 API 调用顺序） + +每个场景都有实际可运行的代码片段、生成的 SQL 预览、返回值输出示例。 + +## 报表引擎管理前端页面 + +为报表引擎创建了可直接使用的前端管理页面，访问 `/admin/report`。 + +### 新增文件 +- `controller/report/report_controller.go`: HTTP API Controller，16个端点暴露公共报表服务的全部能力（业务/报表/字段/抽取配置的 CRUD、数据抽取触发、自动建表、用户选择查询） +- `controller/report/report_admin_controller.go`: 嵌入式HTML SPA，5个Tab页： + - **业务管理**：列表 CRUD + - **报表配置**：按业务筛选列表 CRUD + - **字段配置**：按业务+报表筛选，维度/指标/筛选角色管理 + - **抽取配置**：按业务+报表筛选，支持 DIRECT/AGGREGATE 模式配置 + 一键执行抽取 + - **数据查询**：可视化维度选择器、指标聚合选择器、筛选条件构建器、查询结果展示 +- `main.go`: 注册 reportCtrl.ReportController 到 RouteRegister + `/admin/report` 页面路由 + +### 技术要点 +- 纯原生 HTML/CSS/JS，零框架依赖，与现有管理后台 `/admin` 风格统一 +- 所有 CRUD 通过 `ID *int64` 区分新增（nil）/修改（有值） +- 查询Tab支持动态字段选择：维度多选、指标聚合方式选择、筛选条件动态构建（支持 BETWEEN/IN 等操作符） +- 管理后台页面增加了返回链接 `← 返回管理后台` + +## 修复 report_common_ddl.sql 语法错误 + +PostgreSQL 不支持 MySQL 风格的内联 `COMMENT '...'` 语法。修复了全部5张表的列定义： +- 移除所有列定义中的内联 `COMMENT 'xxx'`（MySQL 语法） +- 已有的 `COMMENT ON COLUMN` 独立语句保留不变 +- `report_extract_log` 表补充了缺失的 `COMMENT ON COLUMN` 语句（business_code/report_code/extract_code/stat_date/extract_type/status/total_count/success_count/fail_count） + +关键区别：MySQL 支持 `column_name VARCHAR(64) COMMENT 'xxx'`，PostgreSQL 必须用 `COMMENT ON COLUMN table.column IS 'xxx'`。 + +## 补充 tenant_id 到报表系统表 + +5张报表系统表原先缺失 `tenant_id`，与项目核心表（`init_core_tables.sql`）不一致。修改内容： +- **DDL SQL** (`sql/report_common_ddl.sql`)：5张表全部新增 `tenant_id BIGINT NOT NULL DEFAULT 0` 列，所有 UNIQUE 约束和索引都加入 tenant_id +- **Go initTables** (`common/report/report.go`)：同步更新 5 张表的 DDL 定义，加上 tenant_id +- 约束变更：`uk_business_report_code UNIQUE (business_code, report_code)` → `(tenant_id, business_code, report_code)`，其余类推 + +当前 report 模块租户支持为占位符级别（硬编码 tenant_id=1），后续需从 context 动态获取。 +- **单独迁移脚本**: `sql/report_add_tenant_id.sql`，对已存在的表执行 ALTER TABLE ADD COLUMN + 重建约束/索引 + +## 快手电商报表初始化种子数据 + +**初版（已废弃）**：字段名随意捏造，与真实 DDL 不对齐。 + +**重写版** `sql/seed_data_report_kuaishou.sql`（基于 seed_data_kuaishou.sql 实际 DDL）： +- 1 条业务配置 (kuaishou_ecommerce) +- 5 条报表配置：order_analysis / item_analysis / refund_analysis / cps_order_analysis / dropshipping_analysis +- 76 条字段配置（订单23+商品15+售后12+分销13+代发13），含维度/指标/衍生字段 +- 5 条抽取配置（AGGREGATE 聚合模式，每个报表一条） + +**严格设计原则**： +- sourceField 对齐源表实际列名（camelCase：oid, status, createTime, totalFee, buyerNick 等） +- targetField / field_code 使用 snake_case（stat_date, total_fee, buyer_nick 等） +- 金额字段源表存分(BIGINT)，报表提供衍生元字段(CALCULATED expression / 100.0) +- 增量抽取依据统一用 updateTime（毫秒时间戳 BIGINT） +- transform_rules：ts_to_date FORMAT createTime → stat_date(yyyy-MM-dd) +- 所有 INSERT 使用 ON CONFLICT DO NOTHING 保证幂等 + +## 修复报表管理前端编辑不显示数据问题 + +在 `report_admin_controller.go` 中修复了编辑弹窗不显示数据的问题： + +**根因**： +- GoFrame `MiddlewareHandlerResponse` 包裹响应为 `{code:0, data: }`，而单条查询返回结构体为 `getBusinessRes{Data: ...}` 序列化为 `{"data": {...}}`，最终响应为 `{code:0, data: {"data": {...}}}` 双层嵌套 +- 某些 GoFrame 版本可能序列化 Go 字段名（首字母大写 `Data`）而非 json tag（`data`），导致 `res.data` 为 undefined + +**修复内容**： +1. `api()` 函数增加兼容逻辑：当 `json.data.Data` 存在而 `json.data.data` 不存在时，自动取 `json.data.Data` +2. `api()` 函数增加 `console.log` 调试日志，打印原始响应和解包后的数据 +3. 四个编辑函数（`openBizModal/openReportModal/openFieldModal/openExtractModal`）： + - 增加 fallback：`res.data !== undefined ? res.data : res.Data` + - 增加数据有效性校验：`if (!d || !d.id)` 时弹出错误提示并阻止打开空表单 + - 增加 `console.log` 调试日志 + +## 修复 report_report_config 查询 SQL 永假条件问题 + +**现象**：查询 `report_report_config` 时生成的 SQL 包含 `WHERE (("id" = 1) AND ("id" = 0 AND "business_code" = '' AND ...))`，id 不可能同时为 1 和 0，导致查询始终返回 0 行。 + +**根因**：`common/db/gfdb/gfdb.go` 中 `Model()` 方法使用了 `OmitNil()`（只过滤 nil 指针，不过滤零值）。当 GoFrame v2 内部的软删除机制（`deleted_at IS NULL`）自动构建 struct WHERE 条件时，由于 `OmitNil()` 不忽略零值字段，结构体中所有字段的零值（`id=0`, `business_code=""`, `report_code=""` 等）都被加入 WHERE 条件，与显式的 `Where("id", 1)` 叠加形成 `id=1 AND id=0` 的永假条件。 + +**修复**：`OmitNil()` **保留**，额外追加 `.OmitEmptyWhere()`，两者同时生效： +- `OmitNil()`：保持 INSERT/UPDATE 的 Data() 行为不变（nil 指针/map 字段不写入 SQL，走 DB 默认值），避免破坏数据引擎写入逻辑 +- `OmitEmptyWhere()`：过滤 struct WHERE 条件中的零值字段，解决软删除 hook 注入零值条件的问题 + +最终代码：`OmitNil().OmitEmptyWhere()` + +**影响范围**：`common/db/gfdb/gfdb.go` - 全局 DB Model 初始化。写入行为完全兼容，查询条件不再包含零值 struct 字段。 + +**安全性审计（2026-06-10）**：逐文件审查了数据引擎核心同步路径全部 40+ 处 `gfdb.Model()` 调用： +- 100% 传字符串表名（非 struct），不会触发 struct WHERE 推断 +- 100% 使用 `Where("col", val)` 字符串键值对（非 struct/map），`OmitEmptyWhere()` 对此无效 +- `OmitEmptyWhere()` 不影响 `Data()`/`Insert()`/`Update()`/`Save()` +- 结论：`OmitEmptyWhere()` 对数据引擎核心同步逻辑完全无影响，唯一生效场景是 `common/report` 软删除 hook + +## 修复 GetXxxByID 方法零值结构体返回问题 + +**现象**：前端编辑报表/业务/字段/抽取配置时一直提示"未获取到XX数据（ID=N）"。与上述 OmitNil 问题同根：SQL 返回 0 行时，GoFrame 的 `One()` 可能不返回 `sql.ErrNoRows`（驱动差异），此时 `GetXxxByID` 判断 `err == nil` 就返回了零值结构体 `{ID: 0, ...}`。Controller 包装为 `{code:0, data:{"data":{id:0}}}`，前端 `!d.id` 判定失败弹出错误提示。 + +**修复**：`common/report/config/loader.go` 中全部 7 个 `One()` 调用方法，在 `err` 检查之外增加 `result.IsEmpty()` 检查： +- `GetBusinessByID` / `GetReportByID` / `GetFieldByID` / `GetExtractConfigByID`：四者模式一致，`_` 改为 `result` 变量并追加 `if result.IsEmpty()` +- `GetBusiness` / `GetReport`（by code）、`GetExtractLog`：同上，追加 `IsEmpty()` 守卫 + +**双重防御**：gfdb.go 的 `OmitEmptyWhere()` 是根本修复（消除错误 SQL）；loader.go 的 `IsEmpty()` 是兜底守卫（即使 SQL 返回 0 行也不返回零值结构体）。 + +## 修复编辑弹窗数据提取的三层兼容 + +**根因**：`api()` 返回 `json.data` 给调用者。当 GoFrame `MiddlewareHandlerResponse` 双层包裹时（`{code:0, data: {"data": {...}}}`），`api()` 返回 `{"data": {...}}`，调用者通过 `res.data` 正确提取。但当 GoFrame 单层包裹时（`{code:0, data: {...}}`），`api()` 直接返回 `{...config...}`，调用者 `res.data` 为 undefined → `d` 为 undefined → 触发 "未获取到XX数据" 错误。 + +**修复**：4 个编辑函数（`openBizModal`/`openReportModal`/`openFieldModal`/`openExtractModal`）的数据提取改为三层 fallback： +```javascript +const d = res && res.data !== undefined ? res.data : (res && res.Data !== undefined ? res.Data : res); +``` +- 第一层：`res.data`（双层包裹，json tag 小写） +- 第二层：`res.Data`（Go 字段名大写 D） +- 第三层：`res` 本身（单层包裹，数据直接在 res 里） + +同时 console.log 增加打印 `raw:` 和 `data:` 便于排查。 + +## 修复报表引擎实体缺失 TenantId 字段 → 查询 tenant_id 不匹配返回零值 + +**现象**：`GET /report/business?id=1` 返回 `code: 0` 但 data 为全零值 `{id: 0, businessCode: "", ...}`，尽管 DB 中实际有数据。 + +**根因**：四个报表引擎实体结构体均缺失 `TenantId` 字段，但 DDL 中表定义有 `tenant_id BIGINT NOT NULL DEFAULT 0`： +1. `CreateBusiness` 序列化 `BusinessConfig` 时，JSON 中无 `tenant_id` → insert map 中无 `tenant_id` key +2. `insertHook` 检查 `in.Data[i]["tenant_id"]` 是否存在 → 不存在 → 不设置 → DB 用 DEFAULT 0 +3. `selectHook` 从 `ctxWithUser` 获取 `TenantId=1`，注入 `WHERE tenant_id = 1` +4. DB 行 `tenant_id=0` 与查询 `tenant_id=1` 不匹配 → 查不到行 → 返回零值结构体 + +**修复**（`common/report/model/model.go`）：给 `BusinessConfig`、`ReportConfig`、`FieldConfig`、`ExtractConfig` 四个结构体新增 `TenantId uint64 \`orm:"tenant_id" json:"tenant_id"\``。 + +**原理**：添加后序列化 JSON 包含 `"tenant_id": 0`，`insertHook` 检测到 key 存在且为零值，自动赋值为 `userInfo.TenantId`。 + +**调试日志**（`common/report/config/loader.go`）：`GetBusinessByID` 增加 `g.Log().Infof` 打印 `id`、`err`、`IsEmpty`、`biz.ID`、`biz.BusinessCode`，便于排查后续类似问题。同时补充 import `"github.com/gogf/gf/v2/frame/g"`。 + +## 修复 GoFrame One(&struct) 直接映射失效 → 改用两步模式 + +**现象**：修复 TenantId 后，SQL 能正确执行并返回1行（`[rows:1]`, `IsEmpty=false`），但 struct 字段全为零值（`biz.ID=0, biz.BusinessCode=""`）。 + +**根因**：`loader.go` 中 7 处 `One(&struct)` 直接传结构体指针，在 GoFrame v2.10.0 + `selectHook` 修改 SQL 的组合下，struct 自动映射失效 —— 查询返回 `gdb.Record` 但未 populate 结构体。 + +**修复**：全部 7 处改为项目已验证的两步模式（参照 `dao/dict/api_interface_dao.go`）： +```go +// 旧写法（struct 映射失效） +result, err := gfdb.DB(ctx).Model(ctx, "table").Where(...).One(&biz) + +// 新写法（先取 Record，再手动 Struct） +r, err := gfdb.DB(ctx).Model(ctx, "table").Where(...).One() +if err = r.Struct(&biz); err != nil { return nil, err } +``` + +**影响的方法**：`GetBusiness`、`GetBusinessByID`、`GetReport`、`GetReportByID`、`GetFieldByID`、`GetExtractConfigByID`、`GetExtractLog`。 diff --git a/.codebuddy/memory/MEMORY.md b/.codebuddy/memory/MEMORY.md index 6a467a9..7bb3815 100644 --- a/.codebuddy/memory/MEMORY.md +++ b/.codebuddy/memory/MEMORY.md @@ -73,11 +73,15 @@ auth_config: { "app_key": "...", "app_secret": "..." } | `cursor_pagination` | bool | 是否游标分页（true=游标，false/无=普通分页） | 通用 | | `time_field` | string | 增量时间字段名 | 通用 | | `time_field_mode` | string | `"range"`=beginTime/endTime模式（快手），`"filtering"`=filtering数组模式（腾讯，默认） | 通用 | +| `full_sync_start_time` | int | 全量同步时的时间起点，毫秒时间戳。`lastSyncTime=0` 时优先读取此值，不存在则回退90天前（range）或不传时间过滤（filtering） | 通用 | | `prefetch` | object | 预取配置（见下方） | 需遍历实体的接口 | +| `recursive` | object | 递归遍历配置（见下方） | 树形结构接口（如钉钉部门） | +| `max_recursive_depth` | int | 递归最大深度，默认20 | 递归遍历使用 | | `body_wrapper_field` | string | 业务参数包装字段名，如`"param"` | 快手 | | `exclude_from_wrapper` | string[] | body_wrapper 时不包装的字段 | 快手 | | `top_level_params` | string[] | 保留在顶层的字段（备用） | 通用 | | `fields` | string[] | API 的 fields 参数 | 腾讯 | +| `row_inject` | string[] | 将请求参数值注入到每行数据中（如 salaryGroupName），解决响应不含某请求参数但需存储的场景 | 通用（prefetch 和非 prefetch 均支持） | ### prefetch 预取配置 ```json @@ -198,6 +202,15 @@ WARN 接口 [kuaishou/order_list] 正在同步中，跳过重复请求 **调度器时序**：使用 `for { runAutoSync(); time.Sleep(interval) }` 模式，而非 `time.NewTicker`。每次同步**完全结束后**才开始计时 interval，避免前一次未跑完就启动下一次导致重叠。无论全量跑 30 分钟还是 70 分钟，下一次都在「本次完成时间 + interval」后执行。 +## 三种遍历模式对比 + +| 模式 | 配置 | 适用场景 | 流程 | +|------|------|---------|------| +| **普通分页** | `page_param` + `page_size_param` | 列表接口分页 | 请求第1页 → 读 `page_info.total_page` → 遍历后续页 | +| **游标分页** | `cursor_pagination: true` | 实时滚动列表（快手订单） | 首次 `cursor=""` → 响应返回 `cursor` → 直到 `""` 或 `"nomore"` | +| **prefetch 预取** | `prefetch: {url, ...}` | 先取实体列表，再查详情 | 分页拉取实体来源列表 → 提取全部 ID → 并发查详情 | +| **recursive 递归** | `recursive: {key_field, target_param}` | 树形结构（钉钉部门） | BFS 队列：根级调用 → 提取子ID → 逐级递归下级 → 防重复 | + ## 新增平台 + 接口的操作步骤 ### 操作步骤 @@ -461,6 +474,69 @@ INSERT INTO api_interface ( ``` 注意：POST 方法且无 `parameters_location: "query"` 时，参数以 JSON body 形式发送。 +#### 模板 E：OAuth2 + POST + 无分页（钉钉风格） +```sql +INSERT INTO api_datasource_platform ( + tenant_id, creator, created_at, updater, updated_at, + platform_code, platform_name, description, status, + api_base_url, auth_type, + auth_config, + rate_limit_per_minute, rate_limit_per_hour, + concurrency_limit, request_timeout_ms, max_retries, retry_delay_ms +) VALUES ( + 1, 'admin', NOW(), 'admin', NOW(), + 'dingtalk', '钉钉', '钉钉开放平台数据同步', 'ACTIVE', + 'https://oapi.dingtalk.com', 'OAUTH2', + '{ + "token_in_query": true, + "query_key": "access_token" + }'::jsonb, + 60, 3600, 5, 30000, 3, 1000 +); + +INSERT INTO api_interface ( + tenant_id, creator, created_at, updater, updated_at, + platform_id, name, code, url, method, status, auth_type, + request_config, response_config, table_definition +) VALUES ( + 1, 'admin', NOW(), 'admin', NOW(), + (SELECT id FROM api_datasource_platform WHERE platform_code = 'dingtalk'), + '{接口名称}', '{接口编码}', + '{相对路径}', 'POST', 'active', 'inherit', + '{ + "parameters_location": "query", + "page_param": "cursor", + "page_size_param": "pageSize", + "{业务参数}": {值} + }'::jsonb, + '{ + "success_field": "errcode", + "success_value": 0, + "message_field": "errmsg", + "list_path": "result" + }'::jsonb, + '{ + "table_name": "{表名}", + "columns": [ + {"name": "id", "type": "BIGINT", "comment": "ID"}, + {"name": "name", "type": "VARCHAR(300)", "comment": "名称"} + ], + "conflict_keys": ["id"] + }'::jsonb +); +``` +注意：钉钉使用 `access_token` 作为 URL 查询参数，通过 `token_in_query: true` + `query_key: "access_token"` 配置。POST 请求通过 `parameters_location: "query"` 将所有参数放在 URL 查询字符串中（DingTalk 同时支持 query 和 body 传参）。 + +如需递归遍历树形结构（如部门全量），在 `request_config` 中添加： +```json +"recursive": { + "key_field": "dept_id", // 从响应行中提取哪个字段的值用于递归 + "target_param": "dept_id" // 递归参数注入到请求中的参数名 +}, +"max_recursive_depth": 20 // 可选，最大递归深度，默认20 +``` +递归流程：BFS 遍历，先查根级空参 → 提取每行的 `key_field` → 逐级注入 `target_param` 查询下级 → 防重复循环保护 → 所有结果合并入库。 + ## 新增接口步骤总结 1. **确定平台**：已有平台直接跳到第3步，否则先新增平台（参考对应认证类型的模板） @@ -474,6 +550,67 @@ INSERT INTO api_interface ( - 是否需要遍历实体（prefetch） - 增量时间字段（字段名 + 值类型） 4. **选择模板**并填入对应值 + + > ⚠️ **全量同步起始时间**：如果接口数据量很大，首次全量不想从 90 天前拉取，可以在 `request_config` 中加 `"full_sync_start_time": <毫秒时间戳>` 指定起点。例如只同步最近 30 天： + > ```json + > "full_sync_start_time": 1745942400000 + > ``` + > - **range 模式**（快手 `time_field_mode: "range"`）：全量时取此时间作为 beginTime + > - **filtering 模式**（腾讯 `time_field_mode: "filtering"`）：全量时生成 `GREATER_EQUALS` 过滤条件 + > - **不配此字段**：行为不变（range 回退 90 天，filtering 不传时间过滤） + 5. **执行 seed SQL** 6. **在管理后台验证**：`http://{host}:3002/admin` +## 代码审计发现的项目规范 + +### 包命名规范（2026-06-03 审计） +- `model/dto/dict/` 下的 `.go` 文件必须用 `package dict`（曾错误写为 `package api_feature`，已修复） +- `model/entity/dict/` 下的 `.go` 文件用 `package dict` +- `consts/api-feature/` 目录下的文件用 `package api_feature` +- `consts/public/` 用 `package public` +- **不要**在 `consts/dict/` 中重复定义 `PlatformStatus`/`ApiMethod` 等类型，应使用 `consts/api-feature` 中的定义 + +### 数据库状态值规范 +- `api_datasource_platform.status`：使用大写 `"ACTIVE"` / `"INACTIVE"` +- `api_interface.status`：使用小写 `"active"` / `"inactive"` +- 代码中查平台状态时不要使用 `api_feature.PlatformStatusActive`（其值为小写），应直接用 `"ACTIVE"` 字符串 + +### Known Bad Practices（已知待改进项） +- 敏感信息（access_token / app_secret）仍分散在 SQL seed文件和 config.yml 中，建议迁移到环境变量或 Vault +- `consts/dict/consts.go` 和 `scheduler/run_sync_task_log_task.go` 可能随业务扩展需要更新 + +## 通用报表引擎 (common/report) + +2026-06-10: 通用报表公共包，配置驱动的报表子系统，支持前端自由选择维度/指标/筛选/时间查询。 + +### 架构分层 (9个Go文件) +``` +common/report/ +├── api.go # ReportService 统一门面（单例） +├── report.go # 5张系统表 DDL +├── example_usage.go # 完整使用示例文档 +├── model/model.go # 实体/请求/响应/常量 +├── config/loader.go # 配置加载器（读缓存+CRUD） +├── builder/sql_builder.go # 动态 SQL 构建 +├── executor/executor.go # 查询执行器 +├── extract/extract.go # 天级数据抽取（DIRECT/AGGREGATE） +└── ddlsync/creator.go # 统计宽表自动创建 +``` + +### 核心能力 +1. **自动建表**：根据 FieldConfig 自动 CREATE TABLE stat_xxx +2. **数据抽取**：DIRECT（逐行）/ AGGREGATE（聚合 SUM/COUNT）模式，幂等保证 +3. **动态查询**：前端选维度+指标+筛选+时间 → 实时构建 SQL → 分页返回 +4. **配置 CRUD**（2026-06-10 新增）：BusinessConfig/ReportConfig/FieldConfig/ExtractConfig 全部可前端维护 + +### 对外接口 (ReportService) +- 查询：`QueryReportByUserSelect` / `GetReportFields` / `GetAllBusinesses` / `GetAllReports` +- 抽取：`ExtractDailyData` / `AutoCreateStatTable` +- CRUD：`SaveBusiness/DeleteBusiness/GetBusiness`、`SaveReport/DeleteReport/GetReport`、`SaveField/DeleteField/GetField`、`SaveExtractConfig/DeleteExtractConfig/GetExtractConfig/GetExtractConfigs` + +### 设计原则 +- 零硬编码：任意平台/接口通过配置接入，不改代码 +- 配置驱动：5张 PG 表存储全部配置，CRUD API 支持前端维护 +- 单例模式：`report.GetService()` 可在任意业务服务中直接调用 + diff --git a/common/report/api.go b/common/report/api.go new file mode 100644 index 0000000..7e69e0d --- /dev/null +++ b/common/report/api.go @@ -0,0 +1,477 @@ +package report + +import ( + "context" + "fmt" + "strings" + + "dataengine/common/report/config" + "dataengine/common/report/ddlsync" + "dataengine/common/report/executor" + "dataengine/common/report/extract" + "dataengine/common/report/model" + + "gitea.redpowerfuture.com/red-future/common/db/gfdb" +) + +// ReportService 报表公共服务 +// 对外暴露的统一接口 +type ReportService struct { + configLoader *config.ConfigLoader + tableCreator *ddlsync.StatTableCreator + queryExecutor *executor.QueryExecutor + dailyExtractor *extract.DailyExtractor +} + +var defaultService *ReportService + +// GetService 获取报表服务单例 +func GetService() *ReportService { + if defaultService == nil { + defaultService = &ReportService{ + configLoader: config.GetLoader(), + tableCreator: ddlsync.NewStatTableCreator(), + queryExecutor: executor.NewQueryExecutor(), + dailyExtractor: extract.NewDailyExtractor(), + } + } + return defaultService +} + +// ============================================================ +// 核心接口 1: 自动创建统计宽表 +// 首次抽取前调用 +// ============================================================ + +// AutoCreateStatTable 根据配置自动创建统计宽表 +// businessCode: 业务编码 +// reportCode: 报表编码 +func (s *ReportService) AutoCreateStatTable(ctx context.Context, businessCode, reportCode string) (*model.AutoCreateStatTableResp, error) { + // 初始化系统表 + if err := initTables(ctx); err != nil { + return nil, fmt.Errorf("初始化系统表失败: %w", err) + } + + resp, err := s.tableCreator.AutoCreateStatTable(ctx, businessCode, reportCode) + if err != nil { + return nil, fmt.Errorf("AutoCreateStatTable 失败: %w", err) + } + + return resp, nil +} + +// ============================================================ +// 核心接口 2: 按天抽取数据 +// 业务层定时任务调用 +// ============================================================ + +// ExtractDailyData 按天抽取数据 +// businessCode: 业务编码 +// reportCode: 报表编码 +// statDate: 统计日期 yyyy-MM-dd +// executor: 执行人 +func (s *ReportService) ExtractDailyData(ctx context.Context, businessCode, reportCode, statDate, executor string) (*model.ExtractDailyDataResp, error) { + // 1. 先确保统计宽表存在 + report, err := s.configLoader.GetReport(ctx, businessCode, reportCode) + if err != nil { + return nil, fmt.Errorf("获取报表配置失败: %w", err) + } + + // 检查表是否存在 + result, err := gfdb.DB(ctx).GetAll(ctx, + "SELECT EXISTS (SELECT 1 FROM pg_tables WHERE tablename = $1) AS exists", + strings.ToLower(report.StatTableName)) + if err != nil { + return nil, fmt.Errorf("检查统计宽表失败: %w", err) + } + tableExists := false + if len(result) > 0 { + tableExists = result[0]["exists"].Bool() + } + if !tableExists { + // 表不存在，先创建 + if _, createErr := s.AutoCreateStatTable(ctx, businessCode, reportCode); createErr != nil { + return nil, fmt.Errorf("创建统计宽表失败: %w", createErr) + } + } + + resp, err := s.dailyExtractor.ExtractDailyData(ctx, businessCode, reportCode, statDate, executor) + if err != nil { + return nil, fmt.Errorf("ExtractDailyData 失败: %w", err) + } + + // 清除缓存 + s.configLoader.InvalidateCache(businessCode, reportCode) + + return resp, nil +} + +// ============================================================ +// 核心接口 3: 用户选择查询（最核心） +// 前端用户选择条件 → 实时构建SQL → 返回报表数据 +// ============================================================ + +// QueryReportByUserSelect 根据用户选择实时查询报表数据 +// 不是自动生成报表，是用户在前端选择维度/指标/筛选/时间后实时查询展示 +func (s *ReportService) QueryReportByUserSelect(ctx context.Context, req *model.UserSelectQueryReq) (*model.UserSelectQueryResp, error) { + // 参数校验 + if req.BusinessCode == "" { + return nil, fmt.Errorf("businessCode 不能为空") + } + if req.ReportCode == "" { + return nil, fmt.Errorf("reportCode 不能为空") + } + + resp, err := s.queryExecutor.QueryReportByUserSelect(ctx, req) + if err != nil { + return nil, fmt.Errorf("QueryReportByUserSelect 失败: %w", err) + } + + return resp, nil +} + +// ============================================================ +// 辅助接口 +// ============================================================ + +// GetReportFields 获取报表可用字段（按维度/指标/筛选分类） +func (s *ReportService) GetReportFields(ctx context.Context, businessCode, reportCode string) (*model.GetReportFieldsResp, error) { + resp, err := s.configLoader.GetReportFields(ctx, businessCode, reportCode) + if err != nil { + return nil, fmt.Errorf("GetReportFields 失败: %w", err) + } + return resp, nil +} + +// GetAllBusinesses 获取所有启用业务列表 +func (s *ReportService) GetAllBusinesses(ctx context.Context) ([]model.BusinessConfig, error) { + return s.configLoader.GetAllBusinesses(ctx) +} + +// GetAllReports 获取业务下所有报表列表 +func (s *ReportService) GetAllReports(ctx context.Context, businessCode string) ([]model.ReportConfig, error) { + return s.configLoader.GetAllReports(ctx, businessCode) +} + +// InvalidateCache 失效指定业务报表缓存 +func (s *ReportService) InvalidateCache(businessCode, reportCode string) { + s.configLoader.InvalidateCache(businessCode, reportCode) +} + +// InitSystemTables 初始化系统表 +func (s *ReportService) InitSystemTables(ctx context.Context) error { + return initTables(ctx) +} + +// ============================================================ +// 配置 CRUD: 业务 +// ============================================================ + +// SaveBusiness 保存业务配置（新增/修改合一） +func (s *ReportService) SaveBusiness(ctx context.Context, req *model.SaveBusinessReq) (*model.SaveResult, error) { + if err := initTables(ctx); err != nil { + return nil, fmt.Errorf("初始化系统表失败: %w", err) + } + + biz := &model.BusinessConfig{ + BusinessCode: req.BusinessCode, + BusinessName: req.BusinessName, + Description: req.Description, + Status: req.Status, + Config: req.Config, + Creator: req.Operator, + Updater: req.Operator, + } + + if req.Status == "" { + biz.Status = model.StatusActive + } + if biz.Config == nil { + biz.Config = make(map[string]interface{}) + } + + if req.ID != nil && *req.ID > 0 { + // 更新 + biz.ID = *req.ID + if err := s.configLoader.UpdateBusiness(ctx, biz); err != nil { + return nil, err + } + return &model.SaveResult{Success: true, ID: *req.ID, Message: "更新成功"}, nil + } + + // 新增 + id, err := s.configLoader.CreateBusiness(ctx, biz) + if err != nil { + return nil, err + } + return &model.SaveResult{Success: true, ID: id, Message: "创建成功"}, nil +} + +// DeleteBusiness 删除业务配置 +func (s *ReportService) DeleteBusiness(ctx context.Context, id int64) (*model.DeleteResult, error) { + biz, err := s.configLoader.GetBusinessByID(ctx, id) + if err != nil { + return nil, err + } + if err := s.configLoader.DeleteBusiness(ctx, id, biz.BusinessCode); err != nil { + return nil, err + } + return &model.DeleteResult{Success: true, Message: "删除成功"}, nil +} + +// GetBusiness 获取单个业务配置 +func (s *ReportService) GetBusiness(ctx context.Context, id int64) (*model.BusinessConfig, error) { + return s.configLoader.GetBusinessByID(ctx, id) +} + +// ============================================================ +// 配置 CRUD: 报表 +// ============================================================ + +// SaveReport 保存报表配置（新增/修改合一） +func (s *ReportService) SaveReport(ctx context.Context, req *model.SaveReportReq) (*model.SaveResult, error) { + if err := initTables(ctx); err != nil { + return nil, fmt.Errorf("初始化系统表失败: %w", err) + } + + rpt := &model.ReportConfig{ + BusinessCode: req.BusinessCode, + ReportCode: req.ReportCode, + ReportName: req.ReportName, + Description: req.Description, + Status: req.Status, + StatTableName: req.StatTableName, + StatTableComment: req.StatTableComment, + DateField: req.DateField, + PrimaryKeys: req.PrimaryKeys, + ConflictKeys: req.ConflictKeys, + Config: req.Config, + Creator: req.Operator, + Updater: req.Operator, + } + + if req.Status == "" { + rpt.Status = model.StatusActive + } + if rpt.DateField == "" { + rpt.DateField = "stat_date" + } + if rpt.PrimaryKeys == nil { + rpt.PrimaryKeys = []string{"id"} + } + if rpt.ConflictKeys == nil { + rpt.ConflictKeys = []string{rpt.DateField} + } + if rpt.Config == nil { + rpt.Config = make(map[string]interface{}) + } + + if req.ID != nil && *req.ID > 0 { + rpt.ID = *req.ID + if err := s.configLoader.UpdateReport(ctx, rpt); err != nil { + return nil, err + } + return &model.SaveResult{Success: true, ID: *req.ID, Message: "更新成功"}, nil + } + + id, err := s.configLoader.CreateReport(ctx, rpt) + if err != nil { + return nil, err + } + return &model.SaveResult{Success: true, ID: id, Message: "创建成功"}, nil +} + +// DeleteReport 删除报表配置 +func (s *ReportService) DeleteReport(ctx context.Context, id int64) (*model.DeleteResult, error) { + rpt, err := s.configLoader.GetReportByID(ctx, id) + if err != nil { + return nil, err + } + if err := s.configLoader.DeleteReport(ctx, id, rpt.BusinessCode, rpt.ReportCode); err != nil { + return nil, err + } + return &model.DeleteResult{Success: true, Message: "删除成功"}, nil +} + +// GetReport 获取单个报表配置 +func (s *ReportService) GetReport(ctx context.Context, id int64) (*model.ReportConfig, error) { + return s.configLoader.GetReportByID(ctx, id) +} + +// ============================================================ +// 配置 CRUD: 字段 +// ============================================================ + +// SaveField 保存字段配置（新增/修改合一） +func (s *ReportService) SaveField(ctx context.Context, req *model.SaveFieldReq) (*model.SaveResult, error) { + if err := initTables(ctx); err != nil { + return nil, fmt.Errorf("初始化系统表失败: %w", err) + } + + field := &model.FieldConfig{ + BusinessCode: req.BusinessCode, + ReportCode: req.ReportCode, + FieldCode: req.FieldCode, + FieldName: req.FieldName, + FieldType: req.FieldType, + DataType: req.DataType, + FieldRole: req.FieldRole, + IsAggregatable: req.IsAggregatable, + IsFilterable: req.IsFilterable, + IsQueryable: req.IsQueryable, + IsSortable: req.IsSortable, + DefaultAggregate: req.DefaultAggregate, + ValidAggregates: req.ValidAggregates, + FilterOperators: req.FilterOperators, + Expression: req.Expression, + ExpressionType: req.ExpressionType, + FormatPattern: req.FormatPattern, + Unit: req.Unit, + DictCode: req.DictCode, + SortOrder: req.SortOrder, + GroupName: req.GroupName, + Status: req.Status, + Creator: req.Operator, + Updater: req.Operator, + } + + if req.Status == "" { + field.Status = model.StatusActive + } + if field.DataType == "" { + field.DataType = model.FieldTypeString + } + if field.ValidAggregates == nil { + field.ValidAggregates = []string{} + } + if field.FilterOperators == nil { + field.FilterOperators = []string{"=", "!=", ">", "<", ">=", "<=", "IN", "LIKE", "BETWEEN"} + } + + if req.ID != nil && *req.ID > 0 { + field.ID = *req.ID + if err := s.configLoader.UpdateField(ctx, field); err != nil { + return nil, err + } + return &model.SaveResult{Success: true, ID: *req.ID, Message: "更新成功"}, nil + } + + id, err := s.configLoader.CreateField(ctx, field) + if err != nil { + return nil, err + } + return &model.SaveResult{Success: true, ID: id, Message: "创建成功"}, nil +} + +// DeleteField 删除字段配置 +func (s *ReportService) DeleteField(ctx context.Context, id int64) (*model.DeleteResult, error) { + field, err := s.configLoader.GetFieldByID(ctx, id) + if err != nil { + return nil, err + } + if err := s.configLoader.DeleteField(ctx, id, field.BusinessCode, field.ReportCode); err != nil { + return nil, err + } + return &model.DeleteResult{Success: true, Message: "删除成功"}, nil +} + +// GetField 获取单个字段配置 +func (s *ReportService) GetField(ctx context.Context, id int64) (*model.FieldConfig, error) { + return s.configLoader.GetFieldByID(ctx, id) +} + +// ============================================================ +// 配置 CRUD: 抽取配置 +// ============================================================ + +// SaveExtractConfig 保存抽取配置（新增/修改合一） +func (s *ReportService) SaveExtractConfig(ctx context.Context, req *model.SaveExtractConfigReq) (*model.SaveResult, error) { + if err := initTables(ctx); err != nil { + return nil, fmt.Errorf("初始化系统表失败: %w", err) + } + + ec := &model.ExtractConfig{ + BusinessCode: req.BusinessCode, + ReportCode: req.ReportCode, + ExtractCode: req.ExtractCode, + ExtractName: req.ExtractName, + SourceTableName: req.SourceTableName, + SourceTableAlias: req.SourceTableAlias, + TargetTableName: req.TargetTableName, + IsEnabled: req.IsEnabled, + ExtractType: req.ExtractType, + ExtractMode: req.ExtractMode, + ExtractKeyField: req.ExtractKeyField, + ExtractKeyFormat: req.ExtractKeyFormat, + GroupByFields: req.GroupByFields, + FilterExpression: req.FilterExpression, + JoinConfigs: req.JoinConfigs, + FieldMappings: req.FieldMappings, + TransformRules: req.TransformRules, + BatchSize: req.BatchSize, + Status: req.Status, + Creator: req.Operator, + Updater: req.Operator, + } + + if req.Status == "" { + ec.Status = model.StatusActive + } + if ec.ExtractType == "" { + ec.ExtractType = model.ExtractTypeIncremental + } + if ec.ExtractMode == "" { + ec.ExtractMode = model.ExtractModeDirect + } + if ec.BatchSize == 0 { + ec.BatchSize = 1000 + } + if ec.JoinConfigs == nil { + ec.JoinConfigs = []model.JoinConfig{} + } + if ec.FieldMappings == nil { + ec.FieldMappings = []model.FieldMapping{} + } + if ec.TransformRules == nil { + ec.TransformRules = []model.TransformRule{} + } + if ec.GroupByFields == nil { + ec.GroupByFields = []string{} + } + + if req.ID != nil && *req.ID > 0 { + ec.ID = *req.ID + if err := s.configLoader.UpdateExtractConfig(ctx, ec); err != nil { + return nil, err + } + return &model.SaveResult{Success: true, ID: *req.ID, Message: "更新成功"}, nil + } + + id, err := s.configLoader.CreateExtractConfig(ctx, ec) + if err != nil { + return nil, err + } + return &model.SaveResult{Success: true, ID: id, Message: "创建成功"}, nil +} + +// DeleteExtractConfig 删除抽取配置 +func (s *ReportService) DeleteExtractConfig(ctx context.Context, id int64) (*model.DeleteResult, error) { + ec, err := s.configLoader.GetExtractConfigByID(ctx, id) + if err != nil { + return nil, err + } + if err := s.configLoader.DeleteExtractConfig(ctx, id, ec.BusinessCode, ec.ReportCode); err != nil { + return nil, err + } + return &model.DeleteResult{Success: true, Message: "删除成功"}, nil +} + +// GetExtractConfig 获取单个抽取配置 +func (s *ReportService) GetExtractConfig(ctx context.Context, id int64) (*model.ExtractConfig, error) { + return s.configLoader.GetExtractConfigByID(ctx, id) +} + +// GetExtractConfigs 获取业务报表下所有抽取配置 +func (s *ReportService) GetExtractConfigs(ctx context.Context, businessCode, reportCode string) ([]model.ExtractConfig, error) { + return s.configLoader.GetExtractConfigs(ctx, businessCode, reportCode) +} diff --git a/common/report/builder/sql_builder.go b/common/report/builder/sql_builder.go new file mode 100644 index 0000000..f219388 --- /dev/null +++ b/common/report/builder/sql_builder.go @@ -0,0 +1,518 @@ +package builder + +import ( + "context" + "fmt" + "regexp" + "strings" + + "dataengine/common/report/config" + "dataengine/common/report/model" +) + +// SQLBuilder 动态SQL构建器 +type SQLBuilder struct { + loader *config.ConfigLoader +} + +// NewSQLBuilder 创建SQL构建器 +func NewSQLBuilder() *SQLBuilder { + return &SQLBuilder{ + loader: config.GetLoader(), + } +} + +// BuildQuerySQL 根据用户选择构建查询SQL +func (b *SQLBuilder) BuildQuerySQL(ctx context.Context, req *model.UserSelectQueryReq) (string, []interface{}, map[string]interface{}, error) { + // 1. 校验配置 + report, err := b.loader.GetReport(ctx, req.BusinessCode, req.ReportCode) + if err != nil { + return "", nil, nil, fmt.Errorf("获取报表配置失败: %w", err) + } + + fieldMap, err := b.loader.GetFieldMap(ctx, req.BusinessCode, req.ReportCode) + if err != nil { + return "", nil, nil, fmt.Errorf("获取字段配置失败: %w", err) + } + + tableName := report.StatTableName + + // 2. 构建 SELECT 部分 + selectClause, err := b.buildSelectClause(req, fieldMap, report) + if err != nil { + return "", nil, nil, err + } + + // 3. 构建 FROM 部分 + fromClause := tableName + + // 4. 构建 WHERE 部分 + whereClause, whereArgs, err := b.buildWhereClause(ctx, req, fieldMap, report) + if err != nil { + return "", nil, nil, err + } + + // 5. 构建 GROUP BY 部分 + groupByClause, err := b.buildGroupByClause(req, fieldMap) + if err != nil { + return "", nil, nil, err + } + + // 6. 构建 ORDER BY 部分 + orderByClause, err := b.buildOrderByClause(req, fieldMap) + if err != nil { + return "", nil, nil, err + } + + // 7. 组合完整SQL + var sql strings.Builder + sql.WriteString("SELECT ") + sql.WriteString(selectClause) + sql.WriteString(" FROM ") + sql.WriteString(fromClause) + + if whereClause != "" { + sql.WriteString(" WHERE ") + sql.WriteString(whereClause) + } + + if groupByClause != "" { + sql.WriteString(" GROUP BY ") + sql.WriteString(groupByClause) + } + + if orderByClause != "" { + sql.WriteString(" ORDER BY ") + sql.WriteString(orderByClause) + } + + // 8. 统计总数SQL + countSql := "SELECT COUNT(*) FROM " + fromClause + if whereClause != "" { + countSql += " WHERE " + whereClause + } + if groupByClause != "" { + countSql = fmt.Sprintf("SELECT COUNT(*) FROM (SELECT 1 FROM %s WHERE %s GROUP BY %s) AS t", + fromClause, whereClause, groupByClause) + } + + metadata := map[string]interface{}{ + "countSql": countSql, + "tableName": tableName, + "reportConfig": report, + } + + return sql.String(), whereArgs, metadata, nil +} + +// buildSelectClause 构建SELECT子句 +func (b *SQLBuilder) buildSelectClause(req *model.UserSelectQueryReq, fieldMap map[string]*model.FieldConfig, report *model.ReportConfig) (string, error) { + var selectParts []string + + // 1. 添加维度字段 + for _, dim := range req.Dimensions { + dim = strings.TrimSpace(dim) + if dim == "" { + continue + } + fc, ok := fieldMap[dim] + if !ok { + return "", fmt.Errorf("维度字段不存在: %s", dim) + } + if fc.FieldRole != model.RoleDimension && fc.FieldRole != model.RoleFilter { + return "", fmt.Errorf("字段 %s 不可作为维度", dim) + } + selectParts = append(selectParts, dim) + } + + // 2. 添加指标字段(含聚合) + if len(req.Indicators) == 0 { + return "", fmt.Errorf("必须选择至少一个指标") + } + + for _, ind := range req.Indicators { + fc, ok := fieldMap[ind.FieldCode] + if !ok { + return "", fmt.Errorf("指标字段不存在: %s", ind.FieldCode) + } + + alias := ind.Alias + if alias == "" { + alias = ind.FieldCode + } + + agg := strings.ToUpper(ind.Aggregate) + if agg == "" { + agg = fc.DefaultAggregate + if agg == "" { + agg = model.AggregateSum + } + } + + // 校验聚合方式 + if len(fc.ValidAggregates) > 0 { + valid := false + for _, v := range fc.ValidAggregates { + if strings.ToUpper(v) == agg { + valid = true + break + } + } + if !valid { + return "", fmt.Errorf("字段 %s 不支持聚合方式 %s", ind.FieldCode, agg) + } + } + + // 处理衍生指标(表达式) + if fc.ExpressionType == "CALCULATED" && fc.Expression != "" { + expr := b.parseExpression(fc.Expression, req.Indicators) + selectParts = append(selectParts, fmt.Sprintf("%s AS %s", expr, alias)) + } else { + selectParts = append(selectParts, fmt.Sprintf("%s(%s) AS %s", agg, ind.FieldCode, alias)) + } + } + + // 3. 添加时间分组字段 + if req.TimeGroup != "" && req.TimeGroup != "day" { + dateField := report.DateField + if dateField == "" { + dateField = "stat_date" + } + timeGroupExpr := b.buildTimeGroupExpr(dateField, req.TimeGroup) + selectParts = append(selectParts, timeGroupExpr) + } + + return strings.Join(selectParts, ", "), nil +} + +// buildWhereClause 构建WHERE子句 +func (b *SQLBuilder) buildWhereClause(ctx context.Context, req *model.UserSelectQueryReq, fieldMap map[string]*model.FieldConfig, report *model.ReportConfig) (string, []interface{}, error) { + var conditions []string + var args []interface{} + + // 1. 租户过滤 + conditions = append(conditions, "tenant_id = 1") + + // 2. 时间范围过滤 + if req.TimeRange != nil { + dateField := report.DateField + if dateField == "" { + dateField = "stat_date" + } + + if req.TimeRange.StartDate != "" { + conditions = append(conditions, fmt.Sprintf("%s >= ?", dateField)) + args = append(args, req.TimeRange.StartDate) + } + if req.TimeRange.EndDate != "" { + conditions = append(conditions, fmt.Sprintf("%s <= ?", dateField)) + args = append(args, req.TimeRange.EndDate) + } + } + + // 3. 业务过滤 + conditions = append(conditions, "business_code = ?") + args = append(args, req.BusinessCode) + + // 4. 用户筛选条件 + for _, filter := range req.Filters { + fc, ok := fieldMap[filter.FieldCode] + if !ok { + return "", nil, fmt.Errorf("筛选字段不存在: %s", filter.FieldCode) + } + + if !fc.IsFilterable { + return "", nil, fmt.Errorf("字段 %s 不可用于筛选", filter.FieldCode) + } + + op := strings.ToUpper(filter.Operator) + if op == "" { + op = "=" + } + + // 校验操作符 + if len(fc.FilterOperators) > 0 { + valid := false + for _, v := range fc.FilterOperators { + if strings.ToUpper(v) == op { + valid = true + break + } + } + if !valid { + return "", nil, fmt.Errorf("字段 %s 不支持操作符 %s", filter.FieldCode, op) + } + } + + cond, vals, err := b.buildFilterCondition(filter, op, fc.FieldType) + if err != nil { + return "", nil, err + } + conditions = append(conditions, cond) + args = append(args, vals...) + } + + return strings.Join(conditions, " AND "), args, nil +} + +// buildFilterCondition 构建单个筛选条件 +func (b *SQLBuilder) buildFilterCondition(filter model.FilterCondition, op string, fieldType string) (string, []interface{}, error) { + field := filter.FieldCode + var args []interface{} + + switch op { + case "=": + return fmt.Sprintf("%s = ?", field), []interface{}{filter.Value}, nil + case "!=": + return fmt.Sprintf("%s != ?", field), []interface{}{filter.Value}, nil + case ">": + return fmt.Sprintf("%s > ?", field), []interface{}{filter.Value}, nil + case "<": + return fmt.Sprintf("%s < ?", field), []interface{}{filter.Value}, nil + case ">=": + return fmt.Sprintf("%s >= ?", field), []interface{}{filter.Value}, nil + case "<=": + return fmt.Sprintf("%s <= ?", field), []interface{}{filter.Value}, nil + case "IN": + values, err := b.convertToSlice(filter.Value) + if err != nil { + return "", nil, err + } + placeholders := make([]string, len(values)) + for i := range values { + placeholders[i] = "?" + args = append(args, values[i]) + } + return fmt.Sprintf("%s IN (%s)", field, strings.Join(placeholders, ",")), args, nil + case "LIKE": + return fmt.Sprintf("%s LIKE ?", field), []interface{}{"%" + fmt.Sprintf("%v", filter.Value) + "%"}, nil + case "BETWEEN": + return fmt.Sprintf("%s BETWEEN ? AND ?", field), []interface{}{filter.Value, filter.Value2}, nil + default: + return fmt.Sprintf("%s = ?", field), []interface{}{filter.Value}, nil + } +} + +// buildGroupByClause 构建GROUP BY子句 +func (b *SQLBuilder) buildGroupByClause(req *model.UserSelectQueryReq, fieldMap map[string]*model.FieldConfig) (string, error) { + var groupFields []string + for _, dim := range req.Dimensions { + fc, ok := fieldMap[dim] + if !ok { + continue + } + if fc.FieldRole == model.RoleDimension || fc.FieldRole == model.RoleFilter { + groupFields = append(groupFields, dim) + } + } + + if len(groupFields) == 0 { + return "", nil + } + + return strings.Join(groupFields, ", "), nil +} + +// buildOrderByClause 构建ORDER BY子句 +func (b *SQLBuilder) buildOrderByClause(req *model.UserSelectQueryReq, fieldMap map[string]*model.FieldConfig) (string, error) { + if len(req.OrderBy) == 0 { + return "", nil + } + + var orderParts []string + for _, order := range req.OrderBy { + field := order.FieldCode + dir := strings.ToUpper(order.Direction) + if dir == "" { + dir = "ASC" + } + if dir != "ASC" && dir != "DESC" { + return "", fmt.Errorf("排序方向必须是 ASC 或 DESC") + } + + fc, ok := fieldMap[field] + if !ok { + return "", fmt.Errorf("排序字段不存在: %s", field) + } + + if !fc.IsSortable { + return "", fmt.Errorf("字段 %s 不可排序", field) + } + + orderParts = append(orderParts, fmt.Sprintf("%s %s", field, dir)) + } + + return strings.Join(orderParts, ", "), nil +} + +// buildTimeGroupExpr 构建时间分组表达式 +func (b *SQLBuilder) buildTimeGroupExpr(dateField, timeGroup string) string { + switch timeGroup { + case "week": + return fmt.Sprintf("DATE_TRUNC('week', %s::date)::text AS time_group", dateField) + case "month": + return fmt.Sprintf("TO_CHAR(%s::date, 'YYYY-MM') AS time_group", dateField) + case "quarter": + return "TO_CHAR(" + dateField + "::date, 'YYYY-\"Q\"Q') AS time_group" + default: + return dateField + " AS time_group" + } +} + +// parseExpression 解析衍生指标表达式 +func (b *SQLBuilder) parseExpression(expr string, indicators []model.IndicatorSelect) string { + re := regexp.MustCompile(`\{([^}]+)\}`) + return re.ReplaceAllStringFunc(expr, func(match string) string { + fieldCode := match[1 : len(match)-1] + for _, ind := range indicators { + if ind.FieldCode == fieldCode { + return fieldCode + } + } + return match + }) +} + +// convertToSlice 转换为切片 +func (b *SQLBuilder) convertToSlice(v interface{}) ([]interface{}, error) { + switch val := v.(type) { + case []interface{}: + return val, nil + case []string: + result := make([]interface{}, len(val)) + for i, s := range val { + result[i] = s + } + return result, nil + case string: + parts := strings.Split(val, ",") + result := make([]interface{}, len(parts)) + for i, p := range parts { + result[i] = strings.TrimSpace(p) + } + return result, nil + default: + return []interface{}{v}, nil + } +} + +// BuildCountSQL 构建统计总数SQL +func (b *SQLBuilder) BuildCountSQL(sql string) string { + sql = regexp.MustCompile(`(?i)SELECT\s+.*?\s+FROM`).ReplaceAllString(sql, "SELECT COUNT(*) FROM") + return sql +} + +// AddLimit 添加分页 +func (b *SQLBuilder) AddLimit(sql string, page, pageSize int) string { + if page < 1 { + page = 1 + } + if pageSize < 1 { + pageSize = 20 + } + if pageSize > 1000 { + pageSize = 1000 + } + offset := (page - 1) * pageSize + return fmt.Sprintf("%s LIMIT %d OFFSET %d", sql, pageSize, offset) +} + +// GenerateInsertSQL 生成upsert SQL +func (b *SQLBuilder) GenerateInsertSQL(tableName string, columns []string, conflictKeys []string) string { + cols := strings.Join(columns, ", ") + placeholders := make([]string, len(columns)) + for i := range columns { + placeholders[i] = fmt.Sprintf("$%d", i+1) + } + placeholdersStr := strings.Join(placeholders, ", ") + + var updateParts []string + for _, col := range columns { + if col == "id" || col == "created_at" { + continue + } + updateParts = append(updateParts, fmt.Sprintf("%s = EXCLUDED.%s", col, col)) + } + + sql := fmt.Sprintf("INSERT INTO %s (%s) VALUES (%s)", tableName, cols, placeholdersStr) + + if len(conflictKeys) > 0 { + sql += " ON CONFLICT (" + strings.Join(conflictKeys, ", ") + ")" + sql += " DO UPDATE SET " + strings.Join(updateParts, ", ") + } + + return sql +} + +// BuildExtractSQL 构建数据抽取SQL +func (b *SQLBuilder) BuildExtractSQL(ctx context.Context, extractConfig *model.ExtractConfig, statDate string) (string, []interface{}, error) { + var selectParts []string + var args []interface{} + + // 基础字段 + selectParts = append(selectParts, "tenant_id") + selectParts = append(selectParts, fmt.Sprintf("'%s' AS business_code", extractConfig.BusinessCode)) + selectParts = append(selectParts, fmt.Sprintf("'%s' AS stat_date", statDate)) + + // 字段映射 + sourceTable := extractConfig.SourceTableName + if extractConfig.SourceTableAlias != "" { + sourceTable = extractConfig.SourceTableAlias + } + + for _, mapping := range extractConfig.FieldMappings { + targetField := mapping.TargetField + sourceField := mapping.SourceField + + if mapping.TransformRule != nil { + expr := b.applyTransformRule(mapping.TransformRule, sourceField) + selectParts = append(selectParts, fmt.Sprintf("%s AS %s", expr, targetField)) + } else { + selectParts = append(selectParts, fmt.Sprintf("%s.%s AS %s", sourceTable, sourceField, targetField)) + } + } + + // 构建 FROM 和 JOIN + fromClause := extractConfig.SourceTableName + if extractConfig.SourceTableAlias != "" { + fromClause += " " + extractConfig.SourceTableAlias + } + + for _, join := range extractConfig.JoinConfigs { + joinType := "LEFT JOIN" + if strings.ToUpper(join.JoinType) == "INNER" { + joinType = "INNER JOIN" + } else if strings.ToUpper(join.JoinType) == "RIGHT" { + joinType = "RIGHT JOIN" + } + fromClause += fmt.Sprintf(" %s %s %s ON %s", joinType, join.JoinTable, join.JoinAlias, join.JoinCondition) + } + + // WHERE 条件 + whereClause := "" + if extractConfig.FilterExpression != "" { + whereClause = " WHERE " + extractConfig.FilterExpression + } + + sql := fmt.Sprintf("SELECT %s FROM %s%s", strings.Join(selectParts, ", "), fromClause, whereClause) + + return sql, args, nil +} + +// applyTransformRule 应用转换规则 +func (b *SQLBuilder) applyTransformRule(rule *model.TransformRule, sourceField string) string { + switch rule.RuleType { + case "CALCULATE": + if rule.Expression != "" { + return strings.ReplaceAll(rule.Expression, "{source}", sourceField) + } + case "FORMAT": + if rule.Format != "" { + return fmt.Sprintf("TO_CHAR(%s, '%s')", sourceField, rule.Format) + } + case "MAPPING": + // 运行时映射，需要在代码中处理 + return sourceField + } + return sourceField +} diff --git a/common/report/config/loader.go b/common/report/config/loader.go new file mode 100644 index 0000000..ac2cc70 --- /dev/null +++ b/common/report/config/loader.go @@ -0,0 +1,664 @@ +package config + +import ( + "context" + "database/sql" + "encoding/json" + "fmt" + "sync" + + "dataengine/common/report/model" + + "gitea.redpowerfuture.com/red-future/common/db/gfdb" + "github.com/gogf/gf/v2/frame/g" +) + +// ConfigLoader 配置加载器 +type ConfigLoader struct { + mu sync.RWMutex + // 缓存 + businessCache map[string]*model.BusinessConfig + reportCache map[string]*model.ReportConfig + fieldCache map[string][]model.FieldConfig + extractCache map[string][]model.ExtractConfig +} + +var ( + defaultLoader *ConfigLoader + once sync.Once +) + +// GetLoader 获取配置加载器单例 +func GetLoader() *ConfigLoader { + once.Do(func() { + defaultLoader = &ConfigLoader{ + businessCache: make(map[string]*model.BusinessConfig), + reportCache: make(map[string]*model.ReportConfig), + fieldCache: make(map[string][]model.FieldConfig), + extractCache: make(map[string][]model.ExtractConfig), + } + }) + return defaultLoader +} + +// GetBusiness 获取业务配置 +func (l *ConfigLoader) GetBusiness(ctx context.Context, businessCode string) (*model.BusinessConfig, error) { + l.mu.RLock() + if biz, ok := l.businessCache[businessCode]; ok { + l.mu.RUnlock() + return biz, nil + } + l.mu.RUnlock() + + var biz model.BusinessConfig + r, err := gfdb.DB(ctx).Model(ctx, "report_business_config"). + Where("business_code", businessCode). + Where("status", model.StatusActive). + One() + if err != nil { + if err == sql.ErrNoRows { + return nil, fmt.Errorf("业务配置不存在: %s", businessCode) + } + return nil, err + } + if r.IsEmpty() { + return nil, fmt.Errorf("业务配置不存在: %s", businessCode) + } + if err = r.Struct(&biz); err != nil { + return nil, err + } + + if biz.Config == nil { + biz.Config = make(map[string]interface{}) + } + + l.mu.Lock() + l.businessCache[businessCode] = &biz + l.mu.Unlock() + + return &biz, nil +} + +// GetReport 获取报表配置 +func (l *ConfigLoader) GetReport(ctx context.Context, businessCode, reportCode string) (*model.ReportConfig, error) { + key := businessCode + ":" + reportCode + l.mu.RLock() + if rpt, ok := l.reportCache[key]; ok { + l.mu.RUnlock() + return rpt, nil + } + l.mu.RUnlock() + + var rpt model.ReportConfig + r, err := gfdb.DB(ctx).Model(ctx, "report_report_config"). + Where("business_code", businessCode). + Where("report_code", reportCode). + Where("status", model.StatusActive). + One() + if err != nil { + if err == sql.ErrNoRows { + return nil, fmt.Errorf("报表配置不存在: %s/%s", businessCode, reportCode) + } + return nil, err + } + if r.IsEmpty() { + return nil, fmt.Errorf("报表配置不存在: %s/%s", businessCode, reportCode) + } + if err = r.Struct(&rpt); err != nil { + return nil, err + } + + if rpt.PrimaryKeys == nil { + rpt.PrimaryKeys = []string{"id"} + } + if rpt.ConflictKeys == nil { + rpt.ConflictKeys = []string{"stat_date"} + } + if rpt.Config == nil { + rpt.Config = make(map[string]interface{}) + } + + l.mu.Lock() + l.reportCache[key] = &rpt + l.mu.Unlock() + + return &rpt, nil +} + +// GetFields 获取报表字段配置 +func (l *ConfigLoader) GetFields(ctx context.Context, businessCode, reportCode string) ([]model.FieldConfig, error) { + key := businessCode + ":" + reportCode + l.mu.RLock() + if fields, ok := l.fieldCache[key]; ok { + l.mu.RUnlock() + return fields, nil + } + l.mu.RUnlock() + + var fields []model.FieldConfig + err := gfdb.DB(ctx).Model(ctx, "report_field_config"). + Where("business_code", businessCode). + Where("report_code", reportCode). + Where("status", model.StatusActive). + Order("sort_order ASC"). + Scan(&fields) + if err != nil { + return nil, err + } + + for i := range fields { + if fields[i].ValidAggregates == nil { + fields[i].ValidAggregates = []string{} + } + if fields[i].FilterOperators == nil { + fields[i].FilterOperators = []string{"=", "!=", ">", "<", ">=", "<=", "IN", "LIKE", "BETWEEN"} + } + } + + l.mu.Lock() + l.fieldCache[key] = fields + l.mu.Unlock() + + return fields, nil +} + +// GetFieldMap 获取字段配置Map +func (l *ConfigLoader) GetFieldMap(ctx context.Context, businessCode, reportCode string) (map[string]*model.FieldConfig, error) { + fields, err := l.GetFields(ctx, businessCode, reportCode) + if err != nil { + return nil, err + } + + fieldMap := make(map[string]*model.FieldConfig) + for i := range fields { + fieldMap[fields[i].FieldCode] = &fields[i] + } + return fieldMap, nil +} + +// GetExtractConfigs 获取抽取配置 +func (l *ConfigLoader) GetExtractConfigs(ctx context.Context, businessCode, reportCode string) ([]model.ExtractConfig, error) { + key := businessCode + ":" + reportCode + l.mu.RLock() + if configs, ok := l.extractCache[key]; ok { + l.mu.RUnlock() + return configs, nil + } + l.mu.RUnlock() + + var configs []model.ExtractConfig + err := gfdb.DB(ctx).Model(ctx, "report_extract_config"). + Where("business_code", businessCode). + Where("report_code", reportCode). + Where("status", model.StatusActive). + Where("is_enabled", true). + Scan(&configs) + if err != nil { + return nil, err + } + + for i := range configs { + if configs[i].JoinConfigs == nil { + configs[i].JoinConfigs = []model.JoinConfig{} + } + if configs[i].FieldMappings == nil { + configs[i].FieldMappings = []model.FieldMapping{} + } + if configs[i].TransformRules == nil { + configs[i].TransformRules = []model.TransformRule{} + } + if configs[i].GroupByFields == nil { + configs[i].GroupByFields = []string{} + } + if configs[i].ExtractMode == "" { + configs[i].ExtractMode = model.ExtractModeDirect + } + } + + l.mu.Lock() + l.extractCache[key] = configs + l.mu.Unlock() + + return configs, nil +} + +// GetExtractLog 获取抽取记录 +func (l *ConfigLoader) GetExtractLog(ctx context.Context, businessCode, reportCode, extractCode, statDate string) (*model.ExtractLog, error) { + var log model.ExtractLog + r, err := gfdb.DB(ctx).Model(ctx, "report_extract_log"). + Where("business_code", businessCode). + Where("report_code", reportCode). + Where("extract_code", extractCode). + Where("stat_date", statDate). + One() + if err != nil { + if err == sql.ErrNoRows { + return nil, nil + } + return nil, err + } + if r.IsEmpty() { + return nil, nil + } + if err = r.Struct(&log); err != nil { + return nil, err + } + return &log, nil +} + +// CreateExtractLog 创建抽取记录 +func (l *ConfigLoader) CreateExtractLog(ctx context.Context, log *model.ExtractLog) error { + data, _ := json.Marshal(log) + var m map[string]interface{} + json.Unmarshal(data, &m) + delete(m, "id") + _, err := gfdb.DB(ctx).Model(ctx, "report_extract_log").Data(m).Save() + return err +} + +// UpdateExtractLog 更新抽取记录 +func (l *ConfigLoader) UpdateExtractLog(ctx context.Context, log *model.ExtractLog) error { + data, _ := json.Marshal(log) + var m map[string]interface{} + json.Unmarshal(data, &m) + delete(m, "id") + delete(m, "created_at") + _, err := gfdb.DB(ctx).Model(ctx, "report_extract_log"). + Where("business_code", log.BusinessCode). + Where("report_code", log.ReportCode). + Where("extract_code", log.ExtractCode). + Where("stat_date", log.StatDate). + Data(m). + Update() + return err +} + +// InvalidateCache 失效缓存 +func (l *ConfigLoader) InvalidateCache(businessCode, reportCode string) { + l.mu.Lock() + delete(l.businessCache, businessCode) + delete(l.reportCache, businessCode+":"+reportCode) + delete(l.fieldCache, businessCode+":"+reportCode) + delete(l.extractCache, businessCode+":"+reportCode) + l.mu.Unlock() +} + +// InvalidateBusinessCache 只失效业务缓存（不影响报表/字段） +func (l *ConfigLoader) InvalidateBusinessCache(businessCode string) { + l.mu.Lock() + delete(l.businessCache, businessCode) + l.mu.Unlock() +} + +// ============================================================ +// CRUD: BusinessConfig +// ============================================================ + +// CreateBusiness 创建业务配置 +func (l *ConfigLoader) CreateBusiness(ctx context.Context, biz *model.BusinessConfig) (int64, error) { + data, _ := json.Marshal(biz) + var m map[string]interface{} + json.Unmarshal(data, &m) + delete(m, "id") + delete(m, "created_at") + delete(m, "updated_at") + delete(m, "deleted_at") + + result, err := gfdb.DB(ctx).Model(ctx, "report_business_config").Data(m).Insert() + if err != nil { + return 0, fmt.Errorf("创建业务配置失败: %w", err) + } + id, _ := result.LastInsertId() + l.InvalidateBusinessCache(biz.BusinessCode) + return id, nil +} + +// UpdateBusiness 更新业务配置 +func (l *ConfigLoader) UpdateBusiness(ctx context.Context, biz *model.BusinessConfig) error { + data, _ := json.Marshal(biz) + var m map[string]interface{} + json.Unmarshal(data, &m) + delete(m, "id") + delete(m, "created_at") + delete(m, "deleted_at") + + _, err := gfdb.DB(ctx).Model(ctx, "report_business_config"). + Where("id", biz.ID). + Data(m). + Update() + if err != nil { + return fmt.Errorf("更新业务配置失败: %w", err) + } + l.InvalidateBusinessCache(biz.BusinessCode) + return nil +} + +// DeleteBusiness 删除业务配置（软删除） +func (l *ConfigLoader) DeleteBusiness(ctx context.Context, id int64, businessCode string) error { + _, err := gfdb.DB(ctx).Model(ctx, "report_business_config"). + Where("id", id). + Data(map[string]interface{}{ + "status": model.StatusInactive, + "deleted_at": "NOW()", + }). + Update() + if err != nil { + return fmt.Errorf("删除业务配置失败: %w", err) + } + l.InvalidateBusinessCache(businessCode) + return nil +} + +// GetBusinessByID 根据ID获取业务配置 +func (l *ConfigLoader) GetBusinessByID(ctx context.Context, id int64) (*model.BusinessConfig, error) { + var biz model.BusinessConfig + r, err := gfdb.DB(ctx).Model(ctx, "report_business_config"). + Where("id", id). + One() + if err != nil { + g.Log().Infof(ctx, "[GetBusinessByID] id=%d, err=%v", id, err) + if err == sql.ErrNoRows { + return nil, fmt.Errorf("业务配置不存在: id=%d", id) + } + return nil, err + } + if r.IsEmpty() { + return nil, fmt.Errorf("业务配置不存在: id=%d", id) + } + if err = r.Struct(&biz); err != nil { + return nil, err + } + g.Log().Infof(ctx, "[GetBusinessByID] id=%d, biz.ID=%d, biz.BusinessCode=%s", + id, biz.ID, biz.BusinessCode) + return &biz, nil +} + +// ============================================================ +// CRUD: ReportConfig +// ============================================================ + +// CreateReport 创建报表配置 +func (l *ConfigLoader) CreateReport(ctx context.Context, rpt *model.ReportConfig) (int64, error) { + data, _ := json.Marshal(rpt) + var m map[string]interface{} + json.Unmarshal(data, &m) + delete(m, "id") + delete(m, "created_at") + delete(m, "updated_at") + delete(m, "deleted_at") + + result, err := gfdb.DB(ctx).Model(ctx, "report_report_config").Data(m).Insert() + if err != nil { + return 0, fmt.Errorf("创建报表配置失败: %w", err) + } + id, _ := result.LastInsertId() + l.InvalidateCache(rpt.BusinessCode, rpt.ReportCode) + return id, nil +} + +// UpdateReport 更新报表配置 +func (l *ConfigLoader) UpdateReport(ctx context.Context, rpt *model.ReportConfig) error { + data, _ := json.Marshal(rpt) + var m map[string]interface{} + json.Unmarshal(data, &m) + delete(m, "id") + delete(m, "created_at") + delete(m, "deleted_at") + + _, err := gfdb.DB(ctx).Model(ctx, "report_report_config"). + Where("id", rpt.ID). + Data(m). + Update() + if err != nil { + return fmt.Errorf("更新报表配置失败: %w", err) + } + l.InvalidateCache(rpt.BusinessCode, rpt.ReportCode) + return nil +} + +// DeleteReport 删除报表配置（软删除） +func (l *ConfigLoader) DeleteReport(ctx context.Context, id int64, businessCode, reportCode string) error { + _, err := gfdb.DB(ctx).Model(ctx, "report_report_config"). + Where("id", id). + Data(map[string]interface{}{ + "status": model.StatusInactive, + "deleted_at": "NOW()", + }). + Update() + if err != nil { + return fmt.Errorf("删除报表配置失败: %w", err) + } + l.InvalidateCache(businessCode, reportCode) + return nil +} + +// GetReportByID 根据ID获取报表配置 +func (l *ConfigLoader) GetReportByID(ctx context.Context, id int64) (*model.ReportConfig, error) { + var rpt model.ReportConfig + r, err := gfdb.DB(ctx).Model(ctx, "report_report_config"). + Where("id", id). + One() + if err != nil { + if err == sql.ErrNoRows { + return nil, fmt.Errorf("报表配置不存在: id=%d", id) + } + return nil, err + } + if r.IsEmpty() { + return nil, fmt.Errorf("报表配置不存在: id=%d", id) + } + if err = r.Struct(&rpt); err != nil { + return nil, err + } + return &rpt, nil +} + +// ============================================================ +// CRUD: FieldConfig +// ============================================================ + +// CreateField 创建字段配置 +func (l *ConfigLoader) CreateField(ctx context.Context, field *model.FieldConfig) (int64, error) { + data, _ := json.Marshal(field) + var m map[string]interface{} + json.Unmarshal(data, &m) + delete(m, "id") + delete(m, "created_at") + delete(m, "updated_at") + delete(m, "deleted_at") + + result, err := gfdb.DB(ctx).Model(ctx, "report_field_config").Data(m).Insert() + if err != nil { + return 0, fmt.Errorf("创建字段配置失败: %w", err) + } + id, _ := result.LastInsertId() + l.InvalidateCache(field.BusinessCode, field.ReportCode) + return id, nil +} + +// UpdateField 更新字段配置 +func (l *ConfigLoader) UpdateField(ctx context.Context, field *model.FieldConfig) error { + data, _ := json.Marshal(field) + var m map[string]interface{} + json.Unmarshal(data, &m) + delete(m, "id") + delete(m, "created_at") + delete(m, "deleted_at") + + _, err := gfdb.DB(ctx).Model(ctx, "report_field_config"). + Where("id", field.ID). + Data(m). + Update() + if err != nil { + return fmt.Errorf("更新字段配置失败: %w", err) + } + l.InvalidateCache(field.BusinessCode, field.ReportCode) + return nil +} + +// DeleteField 删除字段配置（软删除） +func (l *ConfigLoader) DeleteField(ctx context.Context, id int64, businessCode, reportCode string) error { + _, err := gfdb.DB(ctx).Model(ctx, "report_field_config"). + Where("id", id). + Data(map[string]interface{}{ + "status": model.StatusInactive, + "deleted_at": "NOW()", + }). + Update() + if err != nil { + return fmt.Errorf("删除字段配置失败: %w", err) + } + l.InvalidateCache(businessCode, reportCode) + return nil +} + +// GetFieldByID 根据ID获取字段配置 +func (l *ConfigLoader) GetFieldByID(ctx context.Context, id int64) (*model.FieldConfig, error) { + var field model.FieldConfig + r, err := gfdb.DB(ctx).Model(ctx, "report_field_config"). + Where("id", id). + One() + if err != nil { + if err == sql.ErrNoRows { + return nil, fmt.Errorf("字段配置不存在: id=%d", id) + } + return nil, err + } + if r.IsEmpty() { + return nil, fmt.Errorf("字段配置不存在: id=%d", id) + } + if err = r.Struct(&field); err != nil { + return nil, err + } + return &field, nil +} + +// ============================================================ +// CRUD: ExtractConfig +// ============================================================ + +// CreateExtractConfig 创建抽取配置 +func (l *ConfigLoader) CreateExtractConfig(ctx context.Context, ec *model.ExtractConfig) (int64, error) { + data, _ := json.Marshal(ec) + var m map[string]interface{} + json.Unmarshal(data, &m) + delete(m, "id") + delete(m, "created_at") + delete(m, "updated_at") + delete(m, "deleted_at") + + result, err := gfdb.DB(ctx).Model(ctx, "report_extract_config").Data(m).Insert() + if err != nil { + return 0, fmt.Errorf("创建抽取配置失败: %w", err) + } + id, _ := result.LastInsertId() + l.InvalidateCache(ec.BusinessCode, ec.ReportCode) + return id, nil +} + +// UpdateExtractConfig 更新抽取配置 +func (l *ConfigLoader) UpdateExtractConfig(ctx context.Context, ec *model.ExtractConfig) error { + data, _ := json.Marshal(ec) + var m map[string]interface{} + json.Unmarshal(data, &m) + delete(m, "id") + delete(m, "created_at") + delete(m, "deleted_at") + + _, err := gfdb.DB(ctx).Model(ctx, "report_extract_config"). + Where("id", ec.ID). + Data(m). + Update() + if err != nil { + return fmt.Errorf("更新抽取配置失败: %w", err) + } + l.InvalidateCache(ec.BusinessCode, ec.ReportCode) + return nil +} + +// DeleteExtractConfig 删除抽取配置（软删除） +func (l *ConfigLoader) DeleteExtractConfig(ctx context.Context, id int64, businessCode, reportCode string) error { + _, err := gfdb.DB(ctx).Model(ctx, "report_extract_config"). + Where("id", id). + Data(map[string]interface{}{ + "status": model.StatusInactive, + "deleted_at": "NOW()", + }). + Update() + if err != nil { + return fmt.Errorf("删除抽取配置失败: %w", err) + } + l.InvalidateCache(businessCode, reportCode) + return nil +} + +// GetExtractConfigByID 根据ID获取抽取配置 +func (l *ConfigLoader) GetExtractConfigByID(ctx context.Context, id int64) (*model.ExtractConfig, error) { + var ec model.ExtractConfig + r, err := gfdb.DB(ctx).Model(ctx, "report_extract_config"). + Where("id", id). + One() + if err != nil { + if err == sql.ErrNoRows { + return nil, fmt.Errorf("抽取配置不存在: id=%d", id) + } + return nil, err + } + if r.IsEmpty() { + return nil, fmt.Errorf("抽取配置不存在: id=%d", id) + } + if err = r.Struct(&ec); err != nil { + return nil, err + } + return &ec, nil +} + +// GetAllBusinesses 获取所有业务配置 +func (l *ConfigLoader) GetAllBusinesses(ctx context.Context) ([]model.BusinessConfig, error) { + var businesses []model.BusinessConfig + err := gfdb.DB(ctx).Model(ctx, "report_business_config"). + Where("status", model.StatusActive). + Order("id ASC"). + Scan(&businesses) + return businesses, err +} + +// GetAllReports 获取所有报表配置 +func (l *ConfigLoader) GetAllReports(ctx context.Context, businessCode string) ([]model.ReportConfig, error) { + var reports []model.ReportConfig + err := gfdb.DB(ctx).Model(ctx, "report_report_config"). + Where("business_code", businessCode). + Where("status", model.StatusActive). + Order("id ASC"). + Scan(&reports) + return reports, err +} + +// GetReportFields 获取报表可用字段(按角色分类) +func (l *ConfigLoader) GetReportFields(ctx context.Context, businessCode, reportCode string) (*model.GetReportFieldsResp, error) { + fields, err := l.GetFields(ctx, businessCode, reportCode) + if err != nil { + return nil, err + } + + resp := &model.GetReportFieldsResp{ + BusinessCode: businessCode, + ReportCode: reportCode, + Dimensions: []model.FieldConfig{}, + Indicators: []model.FieldConfig{}, + Filters: []model.FieldConfig{}, + } + + for _, f := range fields { + switch f.FieldRole { + case model.RoleDimension: + resp.Dimensions = append(resp.Dimensions, f) + case model.RoleIndicator: + resp.Indicators = append(resp.Indicators, f) + case model.RoleFilter, model.RoleFilterOnly: + resp.Filters = append(resp.Filters, f) + } + } + + return resp, nil +} diff --git a/common/report/ddlsync/creator.go b/common/report/ddlsync/creator.go new file mode 100644 index 0000000..c6a0ab9 --- /dev/null +++ b/common/report/ddlsync/creator.go @@ -0,0 +1,210 @@ +package ddlsync + +import ( + "context" + "fmt" + "strings" + "time" + + "dataengine/common/report/config" + "dataengine/common/report/model" + + "gitea.redpowerfuture.com/red-future/common/db/gfdb" + "github.com/sirupsen/logrus" +) + +// StatTableCreator 统计宽表创建器 +type StatTableCreator struct { + loader *config.ConfigLoader +} + +// NewStatTableCreator 创建统计宽表创建器 +func NewStatTableCreator() *StatTableCreator { + return &StatTableCreator{ + loader: config.GetLoader(), + } +} + +// AutoCreateStatTable 根据配置自动创建统计宽表 +func (c *StatTableCreator) AutoCreateStatTable(ctx context.Context, businessCode, reportCode string) (*model.AutoCreateStatTableResp, error) { + start := time.Now() + logger := logrus.WithFields(logrus.Fields{ + "businessCode": businessCode, + "reportCode": reportCode, + }) + + // 1. 获取报表配置 + report, err := c.loader.GetReport(ctx, businessCode, reportCode) + if err != nil { + return nil, fmt.Errorf("获取报表配置失败: %w", err) + } + + // 2. 获取字段配置 + fields, err := c.loader.GetFields(ctx, businessCode, reportCode) + if err != nil { + return nil, fmt.Errorf("获取字段配置失败: %w", err) + } + + if len(fields) == 0 { + return nil, fmt.Errorf("报表字段配置为空，请先配置字段") + } + + // 3. 构建建表SQL + tableName := report.StatTableName + sql := c.buildCreateTableSQL(tableName, report, fields) + + logger.Infof("创建统计宽表: %s", tableName) + + // 4. 执行建表 + _, err = gfdb.DB(ctx).Exec(ctx, sql) + if err != nil { + return nil, fmt.Errorf("建表失败: %w", err) + } + + // 5. 创建冲突唯一索引 + if len(report.ConflictKeys) > 0 { + indexName := fmt.Sprintf("uq_%s_conflict", tableName) + indexCols := strings.Join(report.ConflictKeys, ", ") + indexSQL := fmt.Sprintf("CREATE UNIQUE INDEX IF NOT EXISTS %s ON %s (%s)", indexName, tableName, indexCols) + if _, err := gfdb.DB(ctx).Exec(ctx, indexSQL); err != nil { + logger.Warnf("创建冲突索引失败: %v", err) + } + } + + // 6. 添加字段注释 + for _, f := range fields { + if f.FieldName != "" { + commentSQL := fmt.Sprintf("COMMENT ON COLUMN %s.%s IS '%s'", tableName, f.FieldCode, strings.ReplaceAll(f.FieldName, "'", "''")) + if _, err := gfdb.DB(ctx).Exec(ctx, commentSQL); err != nil { + logger.Warnf("添加字段注释失败 [%s]: %v", f.FieldCode, err) + } + } + } + + execTime := time.Since(start).Milliseconds() + logger.Infof("统计宽表 %s 创建完成，字段数: %d，耗时: %dms", tableName, len(fields), execTime) + + return &model.AutoCreateStatTableResp{ + Success: true, + TableName: tableName, + ColumnCount: len(fields), + ExecTimeMs: execTime, + }, nil +} + +// buildCreateTableSQL 构建建表SQL +func (c *StatTableCreator) buildCreateTableSQL(tableName string, report *model.ReportConfig, fields []model.FieldConfig) string { + var cols []string + + // 基础字段 + cols = append(cols, "id BIGSERIAL PRIMARY KEY") + cols = append(cols, "tenant_id BIGINT NOT NULL DEFAULT 0") + cols = append(cols, "business_code VARCHAR(64) NOT NULL DEFAULT ''") + cols = append(cols, "creator VARCHAR(64) DEFAULT ''") + cols = append(cols, "created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()") + cols = append(cols, "updater VARCHAR(64) DEFAULT ''") + cols = append(cols, "updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()") + cols = append(cols, "deleted_at TIMESTAMP WITH TIME ZONE") + + // 日期维度字段 + dateField := report.DateField + if dateField == "" { + dateField = "stat_date" + } + cols = append(cols, fmt.Sprintf("%s VARCHAR(16) NOT NULL DEFAULT ''", dateField)) + + // 业务字段 + for _, f := range fields { + colDef := c.fieldTypeToColumn(f.FieldCode, f.FieldType) + cols = append(cols, colDef) + } + + // 原始数据 + cols = append(cols, "raw_data JSONB DEFAULT '{}'") + + sql := fmt.Sprintf("CREATE TABLE IF NOT EXISTS %s (\n %s\n)", tableName, strings.Join(cols, ",\n ")) + + return sql +} + +// fieldTypeToColumn 字段类型转PG列类型 +func (c *StatTableCreator) fieldTypeToColumn(fieldCode, fieldType string) string { + switch fieldType { + case model.FieldTypeInt, model.FieldTypeFloat: + return fmt.Sprintf("%s NUMERIC(20,4) DEFAULT 0", fieldCode) + case model.FieldTypeDate: + return fmt.Sprintf("%s VARCHAR(16) DEFAULT ''", fieldCode) + case model.FieldTypeDatetime: + return fmt.Sprintf("%s TIMESTAMP WITH TIME ZONE", fieldCode) + case model.FieldTypeJsonb: + return fmt.Sprintf("%s JSONB DEFAULT '{}'", fieldCode) + default: // STRING + return fmt.Sprintf("%s VARCHAR(256) DEFAULT ''", fieldCode) + } +} + +// DropStatTable 删除统计宽表 +func (c *StatTableCreator) DropStatTable(ctx context.Context, businessCode, reportCode string) error { + report, err := c.loader.GetReport(ctx, businessCode, reportCode) + if err != nil { + return err + } + + sql := fmt.Sprintf("DROP TABLE IF EXISTS %s CASCADE", report.StatTableName) + _, err = gfdb.DB(ctx).Exec(ctx, sql) + return err +} + +// TableExists 检查表是否存在 +func (c *StatTableCreator) TableExists(ctx context.Context, tableName string) (bool, error) { + result, err := gfdb.DB(ctx).GetAll(ctx, "SELECT COUNT(*) FROM pg_tables WHERE tablename = $1", strings.ToLower(tableName)) + if err != nil { + return false, err + } + if len(result) == 0 { + return false, nil + } + count := result[0]["count"].Int() + return count > 0, nil +} + +// GetTableColumns 获取表字段列表 +func (c *StatTableCreator) GetTableColumns(ctx context.Context, tableName string) ([]string, error) { + var columns []string + sql := `SELECT column_name FROM information_schema.columns WHERE table_name = $1 AND table_schema = 'public' ORDER BY ordinal_position` + rows, err := gfdb.DB(ctx).GetAll(ctx, sql, strings.ToLower(tableName)) + if err != nil { + return nil, err + } + for _, row := range rows.List() { + if col, ok := row["column_name"].(string); ok { + columns = append(columns, col) + } + } + return columns, nil +} + +// AlterTableAddColumns 为已存在的表添加新字段 +func (c *StatTableCreator) AlterTableAddColumns(ctx context.Context, tableName string, newFields []model.FieldConfig) error { + existingCols, err := c.GetTableColumns(ctx, tableName) + if err != nil { + return err + } + + existingMap := make(map[string]bool) + for _, col := range existingCols { + existingMap[col] = true + } + + for _, f := range newFields { + if existingMap[f.FieldCode] { + continue + } + colDef := c.fieldTypeToColumn(f.FieldCode, f.FieldType) + sql := fmt.Sprintf("ALTER TABLE %s ADD COLUMN IF NOT EXISTS %s", tableName, colDef) + if _, err := gfdb.DB(ctx).Exec(ctx, sql); err != nil { + return fmt.Errorf("添加字段 %s 失败: %w", f.FieldCode, err) + } + } + return nil +} diff --git a/common/report/example_usage.go b/common/report/example_usage.go new file mode 100644 index 0000000..3f7cdfd --- /dev/null +++ b/common/report/example_usage.go @@ -0,0 +1,666 @@ +package report + +// ============================================================ +// 通用报表引擎 - 完整调用示例 +// ============================================================ +// +// 包名: dataengine/common/report +// 入口: report.GetService() → *ReportService +// +// 接口一览: +// ┌──────────────┬─────────────────────────────────────────────────┐ +// │ 分类 │ 接口 │ +// ├──────────────┼─────────────────────────────────────────────────┤ +// │ 配置 CRUD │ SaveBusiness / DelBusiness / GetBusiness │ +// │ │ SaveReport / DelReport / GetReport │ +// │ │ SaveField / DelField / GetField │ +// │ │ SaveExtractConfig / DelExtractConfig / Get.. │ +// ├──────────────┼─────────────────────────────────────────────────┤ +// │ 数据抽取 │ ExtractDailyData / AutoCreateStatTable │ +// ├──────────────┼─────────────────────────────────────────────────┤ +// │ 报表查询 │ QueryReportByUserSelect │ +// ├──────────────┼─────────────────────────────────────────────────┤ +// │ 辅助查询 │ GetAllBusinesses / GetAllReports │ +// │ │ GetReportFields / GetExtractConfigs │ +// └──────────────┴─────────────────────────────────────────────────┘ +// +// ============================================================ + +// ============================================================================ +// 场景一：新平台零代码接入（快手电商为例） +// ============================================================================ +// +// 背景：快手订单数据已通过数据引擎同步到 kuaishou_order_list 表（每条订单一行）。 +// 需求：按店铺+天聚合订单数据 → 自动建统计宽表 → 抽取 → 前端自由查询报表。 +// +// 全程只调 API 不写 SQL，前端管理后台可直接操作。 + +/* +package example + +import ( + "context" + "encoding/json" + "fmt" + "time" + + "dataengine/common/report" + "dataengine/common/report/model" +) + +func KuaishouExample() { + ctx := context.Background() + svc := report.GetService() + + // ─── Step 1: 注册业务 ─────────────────────────────────────────── + // 一个业务就是一个数据源平台（快手/抖音/淘宝/...） + _, _ = svc.SaveBusiness(ctx, &model.SaveBusinessReq{ + BusinessCode: "KUAISHOU", // 唯一标识，后续所有接口都用它 + BusinessName: "快手电商", + Description: "快手平台电商业务线", + Status: model.StatusActive, + Operator: "admin", + }) + + // ─── Step 2: 注册报表 ─────────────────────────────────────────── + // 一个业务可以有多个报表（店铺日报、商品日报、主播日报...） + _, _ = svc.SaveReport(ctx, &model.SaveReportReq{ + BusinessCode: "KUAISHOU", + ReportCode: "shop_daily_report", // 报表唯一编码 + ReportName: "快手店铺日报", + StatTableName: "stat_kuaishou_shop_daily", // 统计宽表名（自动创建） + DateField: "stat_date", // 日期字段 + ConflictKeys: []string{"shop_id", "stat_date"}, // 唯一约束（upsert 依据） + Operator: "admin", + }) + + // ─── Step 3: 配置字段（前端可选择的所有维度/指标/筛选） ───────── + // 这是前端"自定义报表"的数据源 —— 前端建好字段后， + // 用户选择哪些维度、哪些指标、怎么筛选，实时查。 + + // 3a. 维度字段：分组依据，不可聚合 + dimensions := []model.SaveFieldReq{ + {BusinessCode: "KUAISHOU", ReportCode: "shop_daily_report", + FieldCode: "shop_id", FieldName: "店铺ID", FieldType: "STRING", + FieldRole: "DIMENSION", IsSortable: true, SortOrder: 1, GroupName: "店铺", Operator: "admin"}, + {BusinessCode: "KUAISHOU", ReportCode: "shop_daily_report", + FieldCode: "shop_name", FieldName: "店铺名称", FieldType: "STRING", + FieldRole: "DIMENSION", IsSortable: true, SortOrder: 2, GroupName: "店铺", Operator: "admin"}, + {BusinessCode: "KUAISHOU", ReportCode: "shop_daily_report", + FieldCode: "stat_date", FieldName: "统计日期", FieldType: "DATE", + FieldRole: "DIMENSION", IsSortable: true, SortOrder: 3, GroupName: "时间", Operator: "admin"}, + } + + // 3b. 指标字段：聚合度量，前端可选 SUM/COUNT/AVG/MAX/MIN + indicators := []model.SaveFieldReq{ + {BusinessCode: "KUAISHOU", ReportCode: "shop_daily_report", + FieldCode: "order_count", FieldName: "订单数", FieldType: "INT", + FieldRole: "INDICATOR", IsAggregatable: true, DefaultAggregate: "SUM", + ValidAggregates: []string{"SUM", "COUNT", "AVG", "MAX", "MIN"}, + SortOrder: 10, GroupName: "订单", Operator: "admin"}, + {BusinessCode: "KUAISHOU", ReportCode: "shop_daily_report", + FieldCode: "order_amount", FieldName: "订单金额(元)", FieldType: "FLOAT", + FieldRole: "INDICATOR", IsAggregatable: true, DefaultAggregate: "SUM", + ValidAggregates: []string{"SUM", "AVG", "MAX", "MIN"}, + SortOrder: 11, GroupName: "金额", Unit: "元", Operator: "admin"}, + {BusinessCode: "KUAISHOU", ReportCode: "shop_daily_report", + FieldCode: "paid_amount", FieldName: "实付金额(元)", FieldType: "FLOAT", + FieldRole: "INDICATOR", IsAggregatable: true, DefaultAggregate: "SUM", + ValidAggregates: []string{"SUM", "AVG"}, + SortOrder: 12, GroupName: "金额", Unit: "元", Operator: "admin"}, + {BusinessCode: "KUAISHOU", ReportCode: "shop_daily_report", + FieldCode: "refund_amount", FieldName: "退款金额(元)", FieldType: "FLOAT", + FieldRole: "INDICATOR", IsAggregatable: true, DefaultAggregate: "SUM", + ValidAggregates: []string{"SUM", "AVG"}, + SortOrder: 13, GroupName: "退款", Unit: "元", Operator: "admin"}, + {BusinessCode: "KUAISHOU", ReportCode: "shop_daily_report", + FieldCode: "buyer_count", FieldName: "下单买家数", FieldType: "INT", + FieldRole: "INDICATOR", IsAggregatable: true, DefaultAggregate: "COUNT", + ValidAggregates: []string{"COUNT"}, + SortOrder: 14, GroupName: "用户", Operator: "admin"}, + // 衍生指标：退款率 = 退款金额 / 订单金额 * 100 + {BusinessCode: "KUAISHOU", ReportCode: "shop_daily_report", + FieldCode: "refund_rate", FieldName: "退款率", FieldType: "FLOAT", + FieldRole: "INDICATOR", IsAggregatable: true, DefaultAggregate: "AVG", + Expression: "{refund_amount} / NULLIF({order_amount}, 0) * 100", + ExpressionType: "CALCULATED", FormatPattern: "#,##0.00", + Unit: "%", SortOrder: 20, GroupName: "退款", Operator: "admin"}, + } + + // 3c. 筛选字段：纯筛选，不出现在 SELECT 中 + filters := []model.SaveFieldReq{ + {BusinessCode: "KUAISHOU", ReportCode: "shop_daily_report", + FieldCode: "order_status", FieldName: "订单状态", FieldType: "STRING", + FieldRole: "FILTER", IsFilterable: true, + FilterOperators: []string{"=", "IN"}, + SortOrder: 30, GroupName: "筛选", Operator: "admin"}, + } + + for _, f := range dimensions { + _, _ = svc.SaveField(ctx, &f) + } + for _, f := range indicators { + _, _ = svc.SaveField(ctx, &f) + } + for _, f := range filters { + _, _ = svc.SaveField(ctx, &f) + } + + // ─── Step 4: 配置数据抽取规则 ──────────────────────────────────── + // 关键：AGGREGATE 模式，从源表聚合到统计宽表 + _, _ = svc.SaveExtractConfig(ctx, &model.SaveExtractConfigReq{ + BusinessCode: "KUAISHOU", + ReportCode: "shop_daily_report", + ExtractCode: "extract_shop_daily", + ExtractName: "快手店铺订单按天聚合", + SourceTableName: "kuaishou_order_list", // ← 源表（订单明细） + SourceTableAlias: "o", + TargetTableName: "stat_kuaishou_shop_daily", // ← 目标（统计宽表） + ExtractType: model.ExtractTypeIncremental, // 增量抽取 + ExtractMode: model.ExtractModeAggregate, // ← 聚合模式 + ExtractKeyField: "created_at", // 增量依据字段 + GroupByFields: []string{"shop_id"}, // ← GROUP BY + FilterExpression: "o.order_status != 'CANCELLED'", // 过滤取消订单 + // 字段映射：源表字段 → 目标表字段 + 聚合函数 + FieldMappings: []model.FieldMapping{ + {SourceField: "shop_id", TargetField: "shop_id", FieldType: "STRING"}, + {SourceField: "shop_name", TargetField: "shop_name", FieldType: "STRING"}, + {SourceField: "id", TargetField: "order_count", FieldType: "INT", AggregateFunction: "COUNT"}, + {SourceField: "order_amount", TargetField: "order_amount", FieldType: "FLOAT", AggregateFunction: "SUM"}, + {SourceField: "paid_amount", TargetField: "paid_amount", FieldType: "FLOAT", AggregateFunction: "SUM"}, + {SourceField: "refund_amount",TargetField: "refund_amount",FieldType: "FLOAT", AggregateFunction: "SUM"}, + {SourceField: "buyer_id", TargetField: "buyer_count", FieldType: "INT", AggregateFunction: "COUNT"}, + }, + BatchSize: 1000, + Operator: "admin", + }) + + // ─── Step 5: 每天定时抽取（cron/k8s CronJob 中调用） ───────── + // ExtractDailyData 内部： + // 1. 检测 stat_kuaishou_shop_daily 表是否存在 + // 2. 不存在 → 根据 FieldConfig 自动 CREATE TABLE + // 3. 从 kuaishou_order_list 按 AGGREGATE 模式抽取当天数据 + // 4. UPSERT 到统计宽表 + today := time.Now().Format("2006-01-02") + resp, _ := svc.ExtractDailyData(ctx, "KUAISHOU", "shop_daily_report", today, "cron") + fmt.Printf("[%s] 抽取完成: 总%d 成功%d 失败%d 耗时%dms\n", + today, resp.TotalCount, resp.SuccessCount, resp.FailCount, resp.ExecTimeMs) + + // 实际抽取生成的 SQL（AGGREGATE 模式）: + // + // INSERT INTO stat_kuaishou_shop_daily (...) + // SELECT ROW_NUMBER() OVER () AS id, + // '2026-06-10' AS stat_date, + // o.shop_id, o.shop_name, + // COUNT(o.id) AS order_count, + // SUM(o.order_amount) AS order_amount, + // SUM(o.paid_amount) AS paid_amount, + // SUM(o.refund_amount) AS refund_amount, + // COUNT(o.buyer_id) AS buyer_count + // FROM kuaishou_order_list o + // WHERE o.created_at::date = '2026-06-10' + // GROUP BY o.shop_id + + // ─── Step 6: 前端查询 ────────────────────────────────────────── + + // 6a. 前端先拉取可用字段列表 + fields, _ := svc.GetReportFields(ctx, "KUAISHOU", "shop_daily_report") + // 返回: + // dimensions: [shop_id, shop_name, stat_date] + // indicators: [order_count, order_amount, paid_amount, refund_amount, buyer_count, refund_rate] + // filters: [order_status] + // 前端据此渲染选择器面板 + + // 6b. 用户选择: 维度=店铺, 指标=金额+订单数, 时间=近7天, 排名=TOP10 + top10Req := &model.UserSelectQueryReq{ + BusinessCode: "KUAISHOU", + ReportCode: "shop_daily_report", + Dimensions: []string{"shop_id", "shop_name"}, + Indicators: []model.IndicatorSelect{ + {FieldCode: "order_amount", Aggregate: "SUM", Alias: "total_amount"}, + {FieldCode: "order_count", Aggregate: "SUM", Alias: "total_orders"}, + {FieldCode: "paid_amount", Aggregate: "SUM", Alias: "total_paid"}, + {FieldCode: "refund_rate", Aggregate: "AVG", Alias: "avg_refund_rate"}, + }, + TimeRange: &model.TimeRange{ + StartDate: time.Now().AddDate(0, 0, -7).Format("2006-01-02"), + EndDate: today, + }, + OrderBy: []model.OrderCondition{{FieldCode: "total_amount", Direction: "DESC"}}, + Page: 1, + PageSize: 10, + } + top10Resp, _ := svc.QueryReportByUserSelect(ctx, top10Req) + fmt.Printf("TOP10 销售额排行 (总数=%d):\n", top10Resp.Total) + for i, row := range top10Resp.List { + b, _ := json.Marshal(row) + fmt.Printf(" %d. %s\n", i+1, string(b)) + } + // 实际生成 SQL: + // SELECT o.shop_id, o.shop_name, + // SUM(o.order_amount) AS total_amount, + // SUM(o.order_count) AS total_orders, + // SUM(o.paid_amount) AS total_paid, + // AVG(refund_amount / NULLIF(order_amount,0) * 100) AS avg_refund_rate + // FROM stat_kuaishou_shop_daily o + // WHERE stat_date BETWEEN '2026-06-03' AND '2026-06-10' + // GROUP BY o.shop_id, o.shop_name + // ORDER BY total_amount DESC + // LIMIT 10 OFFSET 0 + + // 6c. 用户切换维度: 每日趋势（聚合所有店铺） + trendReq := &model.UserSelectQueryReq{ + BusinessCode: "KUAISHOU", + ReportCode: "shop_daily_report", + Dimensions: []string{"stat_date"}, + Indicators: []model.IndicatorSelect{ + {FieldCode: "order_amount", Aggregate: "SUM", Alias: "daily_amount"}, + {FieldCode: "order_count", Aggregate: "SUM", Alias: "daily_orders"}, + {FieldCode: "refund_amount",Aggregate: "SUM", Alias: "daily_refund"}, + }, + TimeRange: &model.TimeRange{ + StartDate: time.Now().AddDate(0, 0, -30).Format("2006-01-02"), + EndDate: today, + }, + TimeGroup: "day", + OrderBy: []model.OrderCondition{{FieldCode: "stat_date", Direction: "ASC"}}, + PageSize: 100, + } + trendResp, _ := svc.QueryReportByUserSelect(ctx, trendReq) + fmt.Printf("30天趋势 (共%d行):\n", trendResp.Total) + + // 6d. 加筛选条件: 只看活跃订单，金额>10000 + filteredReq := &model.UserSelectQueryReq{ + BusinessCode: "KUAISHOU", + ReportCode: "shop_daily_report", + Dimensions: []string{"shop_id", "shop_name"}, + Indicators: []model.IndicatorSelect{ + {FieldCode: "order_amount", Aggregate: "SUM", Alias: "total_amount"}, + }, + Filters: []model.FilterCondition{ + {FieldCode: "order_status", Operator: "=", Value: "ACTIVE"}, + {FieldCode: "total_amount", Operator: ">=", Value: 10000}, // 指标别名也可筛选 + }, + OrderBy: []model.OrderCondition{{FieldCode: "total_amount", Direction: "DESC"}}, + PageSize: 20, + } + _ = filteredReq + + // 6e. 按周汇总趋势 + weeklyReq := &model.UserSelectQueryReq{ + BusinessCode: "KUAISHOU", + ReportCode: "shop_daily_report", + Dimensions: []string{"shop_id"}, + Indicators: []model.IndicatorSelect{ + {FieldCode: "order_amount", Aggregate: "SUM", Alias: "weekly_amount"}, + }, + TimeGroup: "week", + OrderBy: []model.OrderCondition{{FieldCode: "weekly_amount", Direction: "DESC"}}, + PageSize: 50, + } + _ = weeklyReq +} + + +// ============================================================================ +// 场景二：已有配置管理（CRUD 二次开发参考） +// ============================================================================ + +func CRUDExample() { + ctx := context.Background() + svc := report.GetService() + + // ── 业务 CRUD ──────────────────────────────────────────── + + // 新增 + result, _ := svc.SaveBusiness(ctx, &model.SaveBusinessReq{ + BusinessCode: "DOUYIN", BusinessName: "抖音电商", + Operator: "admin", + }) + businessId := result.ID + + // 修改（传 ID 即修改） + result, _ = svc.SaveBusiness(ctx, &model.SaveBusinessReq{ + ID: &businessId, BusinessCode: "DOUYIN", + BusinessName: "抖音电商（新版）", Operator: "admin", + }) + + // 查询 + biz, _ := svc.GetBusiness(ctx, businessId) + fmt.Printf("%s: %s\n", biz.BusinessCode, biz.BusinessName) + + // 删除 + svc.DeleteBusiness(ctx, businessId) + + // 全部列表 + allBiz, _ := svc.GetAllBusinesses(ctx) + for _, b := range allBiz { + fmt.Printf("- %s (%s)\n", b.BusinessCode, b.BusinessName) + } + + // ── 报表 CRUD ──────────────────────────────────────────── + + reportResult, _ := svc.SaveReport(ctx, &model.SaveReportReq{ + BusinessCode: "DOUYIN", + ReportCode: "shop_daily_report", + ReportName: "抖音店铺日报", + StatTableName: "stat_douyin_shop_daily", + ConflictKeys: []string{"shop_id", "stat_date"}, + Operator: "admin", + }) + reportId := reportResult.ID + + rpt, _ := svc.GetReport(ctx, reportId) + svc.DeleteReport(ctx, reportId) + _ = rpt + + // ── 字段 CRUD ──────────────────────────────────────────── + + // 新增字段（id 不传 = 新增） + fieldResult, _ := svc.SaveField(ctx, &model.SaveFieldReq{ + BusinessCode: "DOUYIN", ReportCode: "shop_daily_report", + FieldCode: "order_amount", FieldName: "订单金额", + FieldType: "FLOAT", FieldRole: "INDICATOR", + IsAggregatable: true, DefaultAggregate: "SUM", + ValidAggregates: []string{"SUM", "AVG", "MAX", "MIN"}, + SortOrder: 10, GroupName: "金额", Operator: "admin", + }) + fieldId := fieldResult.ID + + // 修改字段（传 id = 更新） + svc.SaveField(ctx, &model.SaveFieldReq{ + ID: &fieldId, + FieldName: "订单金额（元）", + Operator: "admin", + // ... 只传要修改的字段，未传的保持原值 + }) + + f, _ := svc.GetField(ctx, fieldId) + svc.DeleteField(ctx, fieldId) + _ = f + + // ── 抽取配置 CRUD ──────────────────────────────────────── + + ecResult, _ := svc.SaveExtractConfig(ctx, &model.SaveExtractConfigReq{ + BusinessCode: "DOUYIN", ReportCode: "shop_daily_report", + ExtractCode: "extract_daily", ExtractName: "按天聚合抽取", + SourceTableName: "douyin_order_list", SourceTableAlias: "o", + TargetTableName: "stat_douyin_shop_daily", + ExtractMode: "AGGREGATE", + ExtractKeyField: "created_at", + GroupByFields: []string{"shop_id"}, + FieldMappings: []model.FieldMapping{ + {SourceField: "id", TargetField: "order_count", FieldType: "INT", AggregateFunction: "COUNT"}, + {SourceField: "order_amount", TargetField: "order_amount", FieldType: "FLOAT", AggregateFunction: "SUM"}, + }, + Operator: "admin", + }) + ecId := ecResult.ID + + ec, _ := svc.GetExtractConfig(ctx, ecId) + allEc, _ := svc.GetExtractConfigs(ctx, "DOUYIN", "shop_daily_report") + svc.DeleteExtractConfig(ctx, ecId) + _, _ = ec, allEc +} + + +// ============================================================================ +// 场景三：任意平台接入的通用模式（零硬编码） +// ============================================================================ +// +// 假设要接入淘宝平台，taobao_order_list 表已有数据。 +// 全程不写任何代码，只需调 CRUD API。 + +func GenericPlatformExample() { + ctx := context.Background() + svc := report.GetService() + + // 1. 前端注册业务 + svc.SaveBusiness(ctx, &model.SaveBusinessReq{ + BusinessCode: "TAOBAO", BusinessName: "淘宝电商", + Operator: "admin", + }) + + // 2. 注册报表 + 指定统计宽表名（以后表名不再手工管理） + svc.SaveReport(ctx, &model.SaveReportReq{ + BusinessCode: "TAOBAO", + ReportCode: "shop_daily_report", + ReportName: "淘宝店铺日报", + StatTableName: "stat_taobao_shop_daily", + ConflictKeys: []string{"shop_id", "stat_date"}, + Operator: "admin", + }) + + // 3. 前端选择统计维度（自由组合，随时增删改） + svc.SaveField(ctx, &model.SaveFieldReq{ + BusinessCode: "TAOBAO", ReportCode: "shop_daily_report", + FieldCode: "shop_id", FieldName: "店铺ID", + FieldType: "STRING", FieldRole: "DIMENSION", + SortOrder: 1, GroupName: "店铺", Operator: "admin", + }) + svc.SaveField(ctx, &model.SaveFieldReq{ + BusinessCode: "TAOBAO", ReportCode: "shop_daily_report", + FieldCode: "shop_name", FieldName: "店铺名称", + FieldType: "STRING", FieldRole: "DIMENSION", + SortOrder: 2, GroupName: "店铺", Operator: "admin", + }) + svc.SaveField(ctx, &model.SaveFieldReq{ + BusinessCode: "TAOBAO", ReportCode: "shop_daily_report", + FieldCode: "stat_date", FieldName: "统计日期", + FieldType: "DATE", FieldRole: "DIMENSION", + SortOrder: 3, GroupName: "时间", Operator: "admin", + }) + + // 4. 前端选择统计指标（自由组合，随时增删改） + svc.SaveField(ctx, &model.SaveFieldReq{ + BusinessCode: "TAOBAO", ReportCode: "shop_daily_report", + FieldCode: "order_count", FieldName: "订单数", + FieldType: "INT", FieldRole: "INDICATOR", + IsAggregatable: true, DefaultAggregate: "SUM", + ValidAggregates: []string{"SUM", "COUNT", "AVG"}, + SortOrder: 10, GroupName: "订单", Operator: "admin", + }) + svc.SaveField(ctx, &model.SaveFieldReq{ + BusinessCode: "TAOBAO", ReportCode: "shop_daily_report", + FieldCode: "order_amount", FieldName: "订单金额", + FieldType: "FLOAT", FieldRole: "INDICATOR", + IsAggregatable: true, DefaultAggregate: "SUM", + ValidAggregates: []string{"SUM", "AVG", "MAX", "MIN"}, + SortOrder: 11, GroupName: "金额", Unit: "元", Operator: "admin", + }) + svc.SaveField(ctx, &model.SaveFieldReq{ + BusinessCode: "TAOBAO", ReportCode: "shop_daily_report", + FieldCode: "buyer_count", FieldName: "下单买家数", + FieldType: "INT", FieldRole: "INDICATOR", + IsAggregatable: true, DefaultAggregate: "COUNT", + ValidAggregates: []string{"COUNT"}, + SortOrder: 12, GroupName: "用户", Operator: "admin", + }) + + // 5. 配置抽取规则 + svc.SaveExtractConfig(ctx, &model.SaveExtractConfigReq{ + BusinessCode: "TAOBAO", ReportCode: "shop_daily_report", + ExtractCode: "extract_daily", ExtractName: "淘宝按天聚合", + SourceTableName: "taobao_order_list", SourceTableAlias: "o", + TargetTableName: "stat_taobao_shop_daily", + ExtractMode: "AGGREGATE", + ExtractKeyField: "created_at", + GroupByFields: []string{"shop_id"}, + FieldMappings: []model.FieldMapping{ + {SourceField: "shop_id", TargetField: "shop_id", FieldType: "STRING"}, + {SourceField: "shop_name", TargetField: "shop_name", FieldType: "STRING"}, + {SourceField: "id", TargetField: "order_count", FieldType: "INT", AggregateFunction: "COUNT"}, + {SourceField: "order_amount", TargetField: "order_amount", FieldType: "FLOAT", AggregateFunction: "SUM"}, + {SourceField: "buyer_id", TargetField: "buyer_count", FieldType: "INT", AggregateFunction: "COUNT"}, + }, + BatchSize: 1000, + Operator: "admin", + }) + + // 6. 定时任务每天执行 + svc.ExtractDailyData(ctx, "TAOBAO", "shop_daily_report", "2026-06-10", "cron") + + // 7. 前端实时查询 + req := &model.UserSelectQueryReq{ + BusinessCode: "TAOBAO", ReportCode: "shop_daily_report", + Dimensions: []string{"shop_id", "shop_name"}, + Indicators: []model.IndicatorSelect{ + {FieldCode: "order_amount", Aggregate: "SUM", Alias: "total"}, + {FieldCode: "order_count", Aggregate: "SUM", Alias: "orders"}, + }, + TimeRange: &model.TimeRange{StartDate: "2026-06-01", EndDate: "2026-06-10"}, + Page: 1, PageSize: 20, + } + resp, _ := svc.QueryReportByUserSelect(ctx, req) + fmt.Printf("查询结果: 共%d条, 耗时%dms\n", resp.Total, resp.ExecTimeMs) + + // 平台接入完毕。全程零代码改动。 +} + + +// ============================================================================ +// 场景四：在外部业务服务（如 goview-report）中调用 +// ============================================================================ + +func ExternalServiceExample() { + ctx := context.Background() + svc := report.GetService() + + // 直接用，不需要初始化，内部自动懒加载和建表。 + today := time.Now().Format("2006-01-02") + + // 按天抽取 + svc.ExtractDailyData(ctx, "KUAISHOU", "shop_daily_report", today, "cron") + + // 实时查询 + svc.QueryReportByUserSelect(ctx, &model.UserSelectQueryReq{ + BusinessCode: "KUAISHOU", ReportCode: "shop_daily_report", + Dimensions: []string{"shop_id", "shop_name"}, + Indicators: []model.IndicatorSelect{ + {FieldCode: "order_amount", Aggregate: "SUM", Alias: "total_amount"}, + }, + TimeRange: &model.TimeRange{StartDate: "2026-06-01", EndDate: today}, + Page: 1, PageSize: 20, + }) +} + + +// ============================================================================ +// 场景五：Direct 模式（逐行抽取，不做聚合） +// ============================================================================ +// +// 适用于源表已经是一行一条统计记录的场景（如已预聚合的报表源表）。 + +func DirectModeExample() { + ctx := context.Background() + svc := report.GetService() + + svc.SaveBusiness(ctx, &model.SaveBusinessReq{ + BusinessCode: "HDWL", BusinessName: "HDWL业务", Operator: "admin", + }) + svc.SaveReport(ctx, &model.SaveReportReq{ + BusinessCode: "HDWL", ReportCode: "shop_daily_stat", + ReportName: "HDWL店铺日报", StatTableName: "stat_hdwl_shop_daily", + ConflictKeys: []string{"report_date"}, + Operator: "admin", + }) + + // Direct 模式：不聚合，逐行映射 + svc.SaveExtractConfig(ctx, &model.SaveExtractConfigReq{ + BusinessCode: "HDWL", ReportCode: "shop_daily_stat", + ExtractCode: "extract_direct", ExtractName: "直接逐行抽取", + SourceTableName: "hdwl_daily_summary", SourceTableAlias: "s", + TargetTableName: "stat_hdwl_shop_daily", + ExtractMode: "DIRECT", // ← DIRECT 模式 + ExtractKeyField: "report_date", + FieldMappings: []model.FieldMapping{ + {SourceField: "shop_id", TargetField: "shop_id", FieldType: "STRING"}, + {SourceField: "shop_name", TargetField: "shop_name", FieldType: "STRING"}, + {SourceField: "total_sale", TargetField: "sale_amount", FieldType: "FLOAT"}, + {SourceField: "total_orders", TargetField: "order_count", FieldType: "INT"}, + }, + Operator: "admin", + }) + + svc.ExtractDailyData(ctx, "HDWL", "shop_daily_stat", "2026-06-10", "cron") +} + + +// ============================================================================ +// 场景六：前端交互流程参考 +// ============================================================================ +// +// 前端代码调用示例（伪代码，展示 API 调用顺序）： +// +// // 1. 页面加载 → 获取业务列表 → 渲染下拉框 +// GET /api/report/businesses +// 返回: [{ businessCode: "KUAISHOU", businessName: "快手电商" }, ...] +// +// // 2. 用户选择业务 → 获取报表列表 +// GET /api/report/reports?businessCode=KUAISHOU +// 返回: [{ reportCode: "shop_daily_report", reportName: "快手店铺日报" }, ...] +// +// // 3. 用户选择报表 → 获取可用字段 → 渲染维度/指标/筛选选择器 +// GET /api/report/fields?businessCode=KUAISHOU&reportCode=shop_daily_report +// 返回: { dimensions: [...], indicators: [...], filters: [...] } +// +// // 4. 用户选好条件 → 查询 +// POST /api/report/query +// Body: { +// "businessCode": "KUAISHOU", "reportCode": "shop_daily_report", +// "dimensions": ["shop_id", "shop_name"], +// "indicators": [ +// {"fieldCode": "order_amount", "aggregate": "SUM", "alias": "total"}, +// {"fieldCode": "order_count", "aggregate": "SUM", "alias": "count"} +// ], +// "timeRange": {"startDate": "2026-06-01", "endDate": "2026-06-10"}, +// "orderBy": [{"fieldCode": "total", "direction": "DESC"}], +// "page": 1, "pageSize": 20 +// } +// 返回: { list: [...], total: 152, page: 1, totalPages: 8, execTimeMs: 45 } +// +// // 5. 翻页/换维度/换指标 → 重新调 Step 4 +// +// 管理后台（用户可自行维护配置）： +// +// // 新增业务 +// POST /api/report/business/save +// Body: { "businessCode": "TAOBAO", "businessName": "淘宝电商" } +// +// // 新增/修改字段（用户自定义统计维度） +// POST /api/report/field/save +// Body: { "businessCode": "TAOBAO", "reportCode": "shop_daily_report", +// "fieldCode": "category", "fieldName": "商品类目", +// "fieldType": "STRING", "fieldRole": "DIMENSION" } +// +// // 配置抽取规则 +// POST /api/report/extractConfig/save +// Body: { "businessCode": "TAOBAO", "reportCode": "shop_daily_report", +// "extractCode": "extract_daily", "sourceTableName": "taobao_order_list", +// "extractMode": "AGGREGATE", "groupByFields": ["shop_id"], +// "fieldMappings": [...] } + + +// ============================================================================ +// Direct vs AGGREGATE 模式对比 +// ============================================================================ +// +// ┌──────────┬──────────────────────────────────────────────┐ +// │ 模式 │ 行为 │ +// ├──────────┼──────────────────────────────────────────────┤ +// │ DIRECT │ 逐行映射，1:1 从源表复制到目标表 │ +// │ │ 适用：源表已是一行一统计 │ +// │ │ SQL: SELECT ... FROM source │ +// ├──────────┼──────────────────────────────────────────────┤ +// │ AGGREGATE│ GROUP BY + 聚合函数，N:1 聚合 │ +// │ │ 适用：源表是明细表（如订单行） │ +// │ │ SQL: SELECT ... SUM() COUNT() ... │ +// │ │ FROM source GROUP BY groupByFields │ +// └──────────┴──────────────────────────────────────────────┘ +// +// 可聚合函数: SUM / COUNT / AVG / MAX / MIN +// 字段角色: DIMENSION(维度) / INDICATOR(指标) / FILTER(筛选) / FILTER_ONLY +// 字段类型: STRING / INT / FLOAT / DATE / DATETIME / JSONB +// 操作符: = / != / > / < / >= / <= / IN / LIKE / BETWEEN +// 时间分组: day / week / month / quarter +*/ diff --git a/common/report/executor/executor.go b/common/report/executor/executor.go new file mode 100644 index 0000000..351da8f --- /dev/null +++ b/common/report/executor/executor.go @@ -0,0 +1,192 @@ +package executor + +import ( + "context" + "fmt" + "math" + "strings" + "time" + + "dataengine/common/report/builder" + "dataengine/common/report/model" + + "gitea.redpowerfuture.com/red-future/common/db/gfdb" + "github.com/sirupsen/logrus" +) + +// QueryExecutor SQL执行器 +type QueryExecutor struct { + sqlBuilder *builder.SQLBuilder +} + +// NewQueryExecutor 创建查询执行器 +func NewQueryExecutor() *QueryExecutor { + return &QueryExecutor{ + sqlBuilder: builder.NewSQLBuilder(), + } +} + +// QueryReportByUserSelect 根据用户选择实时查询报表数据（核心接口） +func (e *QueryExecutor) QueryReportByUserSelect(ctx context.Context, req *model.UserSelectQueryReq) (*model.UserSelectQueryResp, error) { + start := time.Now() + logger := logrus.WithFields(logrus.Fields{ + "businessCode": req.BusinessCode, + "reportCode": req.ReportCode, + }) + + // 1. 参数校验 + if err := e.validateReq(req); err != nil { + return nil, fmt.Errorf("参数校验失败: %w", err) + } + + // 2. 构建查询SQL + querySQL, queryArgs, metadata, err := e.sqlBuilder.BuildQuerySQL(ctx, req) + if err != nil { + return nil, fmt.Errorf("构建查询SQL失败: %w", err) + } + + logger.Debugf("查询SQL: %s args: %v", querySQL, queryArgs) + + // 3. 获取记录总数 + total, err := e.executeCount(ctx, metadata, queryArgs) + if err != nil { + return nil, fmt.Errorf("查询总数失败: %w", err) + } + + // 4. 分页查询 + if req.Page < 1 { + req.Page = 1 + } + if req.PageSize < 1 { + req.PageSize = 20 + } + if req.PageSize > 1000 { + req.PageSize = 1000 + } + + pagedSQL := e.sqlBuilder.AddLimit(querySQL, req.Page, req.PageSize) + logger.Debugf("分页SQL: %s", pagedSQL) + + dataResult, err := gfdb.DB(ctx).GetAll(ctx, pagedSQL, queryArgs...) + if err != nil { + return nil, fmt.Errorf("查询数据失败: %w", err) + } + + var list []map[string]interface{} + if dataResult.Len() > 0 { + list = dataResult.List() + } + + // 计算总页数 + totalPages := int(math.Ceil(float64(total) / float64(req.PageSize))) + + execTime := time.Since(start).Milliseconds() + + resp := &model.UserSelectQueryResp{ + List: list, + Total: total, + Page: req.Page, + PageSize: req.PageSize, + TotalPages: totalPages, + Sql: querySQL, + ExecTimeMs: execTime, + } + + logger.Infof("报表查询完成, 总数:%d 耗时:%dms", total, execTime) + + return resp, nil +} + +// executeCount 执行总数统计 +func (e *QueryExecutor) executeCount(ctx context.Context, metadata map[string]interface{}, args []interface{}) (int64, error) { + countSql, ok := metadata["countSql"].(string) + if !ok || countSql == "" { + return 0, nil + } + + result, err := gfdb.DB(ctx).GetAll(ctx, countSql, args...) + if err != nil { + return 0, fmt.Errorf("统计总数失败: %w", err) + } + + var total int64 + if result.Len() > 0 { + for k, v := range result.List()[0] { + _ = k + if n, ok := v.(int64); ok { + total = n + } else { + total = result[0]["count"].Int64() + } + break + } + } + return total, nil +} + +// validateReq 校验请求参数 +func (e *QueryExecutor) validateReq(req *model.UserSelectQueryReq) error { + if req.BusinessCode == "" { + return fmt.Errorf("业务编码不能为空") + } + if req.ReportCode == "" { + return fmt.Errorf("报表编码不能为空") + } + if len(req.Indicators) == 0 { + return fmt.Errorf("必须选择至少一个指标") + } + + // 校验指标 + for i, ind := range req.Indicators { + if ind.FieldCode == "" { + return fmt.Errorf("第%d个指标字段编码不能为空", i+1) + } + if ind.Aggregate != "" { + agg := strings.ToUpper(ind.Aggregate) + validAggs := map[string]bool{ + "SUM": true, + "COUNT": true, + "AVG": true, + "MAX": true, + "MIN": true, + } + if !validAggs[agg] { + return fmt.Errorf("不支持的聚合方式: %s", ind.Aggregate) + } + } + } + + // 校验时间分组 + if req.TimeGroup != "" { + validGroups := map[string]bool{ + "day": true, + "week": true, + "month": true, + "quarter": true, + } + if !validGroups[req.TimeGroup] { + return fmt.Errorf("不支持的时间分组: %s", req.TimeGroup) + } + } + + return nil +} + +// RawQuery 执行原始SQL查询（用于特殊场景） +func (e *QueryExecutor) RawQuery(ctx context.Context, sql string, args ...interface{}) ([]map[string]interface{}, error) { + result, err := gfdb.DB(ctx).GetAll(ctx, sql, args...) + if err != nil { + return nil, fmt.Errorf("原始查询失败: %w", err) + } + + if result.Len() > 0 { + return result.List(), nil + } + return nil, nil +} + +// ExecRaw 执行原始SQL（INSERT/UPDATE/DELETE） +func (e *QueryExecutor) ExecRaw(ctx context.Context, sql string, args ...interface{}) error { + _, err := gfdb.DB(ctx).Exec(ctx, sql, args...) + return err +} diff --git a/common/report/extract/extract.go b/common/report/extract/extract.go new file mode 100644 index 0000000..2d3e9df --- /dev/null +++ b/common/report/extract/extract.go @@ -0,0 +1,644 @@ +package extract + +import ( + "context" + "fmt" + "strings" + "time" + + "dataengine/common/report/config" + "dataengine/common/report/model" + + "gitea.redpowerfuture.com/red-future/common/db/gfdb" + "github.com/gogf/gf/v2/util/gconv" + "github.com/sirupsen/logrus" +) + +// DailyExtractor 天级数据抽取器 +type DailyExtractor struct { + loader *config.ConfigLoader +} + +// NewDailyExtractor 创建抽取器 +func NewDailyExtractor() *DailyExtractor { + return &DailyExtractor{ + loader: config.GetLoader(), + } +} + +// ExtractDailyData 按天抽取数据（业务层定时任务调用） +func (e *DailyExtractor) ExtractDailyData(ctx context.Context, businessCode, reportCode, statDate, executor string) (*model.ExtractDailyDataResp, error) { + start := time.Now() + logger := logrus.WithFields(logrus.Fields{ + "businessCode": businessCode, + "reportCode": reportCode, + "statDate": statDate, + }) + + // 1. 获取报表配置 + report, err := e.loader.GetReport(ctx, businessCode, reportCode) + if err != nil { + return nil, fmt.Errorf("获取报表配置失败: %w", err) + } + + // 2. 获取抽取配置 + extractConfigs, err := e.loader.GetExtractConfigs(ctx, businessCode, reportCode) + if err != nil { + return nil, fmt.Errorf("获取抽取配置失败: %w", err) + } + + if len(extractConfigs) == 0 { + return nil, fmt.Errorf("没有可用的抽取配置") + } + + // 3. 获取字段配置 + fieldMap, err := e.loader.GetFieldMap(ctx, businessCode, reportCode) + if err != nil { + return nil, fmt.Errorf("获取字段配置失败: %w", err) + } + + // 4. 确保统计宽表存在 + if err := e.ensureStatTableExists(ctx, report, fieldMap); err != nil { + return nil, fmt.Errorf("确保统计宽表存在失败: %w", err) + } + + totalCount := 0 + successCount := 0 + failCount := 0 + var lastErr error + + // 5. 遍历每个抽取配置 + for _, ec := range extractConfigs { + // 检查幂等性 + exLog, err := e.loader.GetExtractLog(ctx, businessCode, reportCode, ec.ExtractCode, statDate) + if err != nil { + logger.Errorf("获取抽取记录失败: %v", err) + } + + if exLog != nil && exLog.Status == model.ExtractStatusSuccess { + logger.Infof("抽取配置 %s 日期 %s 已完成，跳过", ec.ExtractCode, statDate) + continue + } + + // 创建抽取记录 + extractLog := &model.ExtractLog{ + BusinessCode: businessCode, + ReportCode: reportCode, + ExtractCode: ec.ExtractCode, + StatDate: statDate, + ExtractType: ec.ExtractType, + Status: model.ExtractStatusRunning, + Executor: executor, + StartTime: &start, + } + _ = e.loader.CreateExtractLog(ctx, extractLog) + + // 执行抽取 + c, s, f, err := e.executeExtract(ctx, &ec, report, fieldMap, statDate) + totalCount += c + successCount += s + failCount += f + + // 更新抽取记录 + now := time.Now() + extractLog.EndTime = &now + extractLog.TotalCount = c + extractLog.SuccessCount = s + extractLog.FailCount = f + + if err != nil { + extractLog.Status = model.ExtractStatusFailed + extractLog.ErrorMessage = err.Error() + lastErr = err + logger.Errorf("抽取配置 %s 执行失败: %v", ec.ExtractCode, err) + } else { + extractLog.Status = model.ExtractStatusSuccess + logger.Infof("抽取配置 %s 完成, 总数:%d 成功:%d 失败:%d", ec.ExtractCode, c, s, f) + } + + if updateErr := e.loader.UpdateExtractLog(ctx, extractLog); updateErr != nil { + logger.Errorf("更新抽取记录失败: %v", updateErr) + } + } + + execTime := time.Since(start).Milliseconds() + logger.Infof("按天抽取完成, 总数:%d 成功:%d 失败:%d 耗时:%dms", totalCount, successCount, failCount, execTime) + + resp := &model.ExtractDailyDataResp{ + Success: lastErr == nil, + TotalCount: totalCount, + SuccessCount: successCount, + FailCount: failCount, + ExecTimeMs: execTime, + } + + if lastErr != nil { + resp.ErrorMsg = lastErr.Error() + } + + return resp, nil +} + +// executeExtract 执行单个抽取配置 +func (e *DailyExtractor) executeExtract(ctx context.Context, ec *model.ExtractConfig, report *model.ReportConfig, fieldMap map[string]*model.FieldConfig, statDate string) (total, success, fail int, err error) { + logger := logrus.WithField("extractCode", ec.ExtractCode) + + // 1. 构建抽取SQL + extractSQL, whereArgs, err := e.buildExtractSQL(ctx, ec, report, statDate) + if err != nil { + return 0, 0, 0, fmt.Errorf("构建抽取SQL失败: %w", err) + } + + logger.Debugf("抽取SQL: %s", extractSQL) + + // 2. 分批抽取 + batchSize := ec.BatchSize + if batchSize <= 0 { + batchSize = 1000 + } + + offset := 0 + for { + // 添加分页 + pagedSQL := fmt.Sprintf("%s LIMIT %d OFFSET %d", extractSQL, batchSize, offset) + args := append(whereArgs) + + rows, queryErr := gfdb.DB(ctx).GetAll(ctx, pagedSQL, args...) + if queryErr != nil { + return total, success, fail, fmt.Errorf("抽取查询失败: %w", queryErr) + } + + batchCount := rows.Len() + if batchCount == 0 { + break + } + + // 3. 应用转换规则（仅 DIRECT 模式需注入审计字段，AGGREGATE 模式已由SQL处理） + dataList := rows.List() + if ec.ExtractMode != model.ExtractModeAggregate { + for i := range dataList { + e.applyTransformRules(ec, dataList[i]) + dataList[i]["tenant_id"] = 1 + dataList[i]["business_code"] = ec.BusinessCode + } + } + + // 4. 写入统计宽表 + c, _, writeErr := e.batchUpsert(ctx, report.StatTableName, report.ConflictKeys, dataList) + if writeErr != nil { + logger.Errorf("批量写入失败 (offset=%d): %v", offset, writeErr) + fail += batchCount + } else { + success += c + } + + total += batchCount + offset += batchSize + + if batchCount < batchSize { + break + } + } + + return total, success, fail, nil +} + +// buildExtractSQL 构建抽取SQL +func (e *DailyExtractor) buildExtractSQL(ctx context.Context, ec *model.ExtractConfig, report *model.ReportConfig, statDate string) (string, []interface{}, error) { + var args []interface{} + + sourceTable := ec.SourceTableName + if ec.SourceTableAlias != "" { + sourceTable = ec.SourceTableAlias + } else { + sourceTable = "s" + } + + // 日期字段 + dateField := report.DateField + if dateField == "" { + dateField = "stat_date" + } + + // 判断抽取模式 + mode := ec.ExtractMode + if mode == "" { + mode = model.ExtractModeDirect + } + + if mode == model.ExtractModeAggregate { + return e.buildAggregateExtractSQL(ec, report, sourceTable, dateField, statDate) + } + + // === 默认 DIRECT 模式：逐行抽取 === + return e.buildDirectExtractSQL(ec, report, sourceTable, dateField, statDate), args, nil +} + +// buildDirectExtractSQL 逐行抽取模式SQL（直接映射，不做聚合） +func (e *DailyExtractor) buildDirectExtractSQL(ec *model.ExtractConfig, report *model.ReportConfig, sourceTable, dateField, statDate string) string { + var selectParts []string + + // 基础审计字段（常量注入） + selectParts = append(selectParts, "0 AS id") + selectParts = append(selectParts, "1 AS tenant_id") + selectParts = append(selectParts, fmt.Sprintf("'%s' AS business_code", ec.BusinessCode)) + selectParts = append(selectParts, "'system' AS creator") + selectParts = append(selectParts, "NOW() AS created_at") + selectParts = append(selectParts, "'system' AS updater") + selectParts = append(selectParts, "NOW() AS updated_at") + selectParts = append(selectParts, "NULL::TIMESTAMP AS deleted_at") + + // 日期字段 + selectParts = append(selectParts, fmt.Sprintf("'%s' AS %s", statDate, dateField)) + + // 原始数据 + selectParts = append(selectParts, "'{}'::JSONB AS raw_data") + + // 字段映射 + for _, mapping := range ec.FieldMappings { + targetField := mapping.TargetField + sourceField := mapping.SourceField + + var expr string + if mapping.TransformRule != nil { + expr = e.applyTransformExpr(mapping.TransformRule, fmt.Sprintf("%s.%s", sourceTable, sourceField)) + } else { + expr = fmt.Sprintf("%s.%s", sourceTable, sourceField) + } + + if mapping.DefaultValue != nil { + expr = fmt.Sprintf("COALESCE(%s, '%v')", expr, mapping.DefaultValue) + } + + selectParts = append(selectParts, fmt.Sprintf("%s AS %s", expr, targetField)) + } + + // FROM + JOIN + fromClause := e.buildFromClause(ec, sourceTable) + + // JOIN 字段映射 + selectParts = append(selectParts, e.buildJoinFieldSelects(ec)...) + + // WHERE + whereClause := e.buildWhereClause(ec, sourceTable, statDate) + + return fmt.Sprintf("SELECT %s FROM %s %s", strings.Join(selectParts, ", "), fromClause, whereClause) +} + +// buildAggregateExtractSQL 聚合抽取模式SQL（GROUP BY + SUM/COUNT/AVG） +func (e *DailyExtractor) buildAggregateExtractSQL(ec *model.ExtractConfig, report *model.ReportConfig, sourceTable, dateField, statDate string) (string, []interface{}, error) { + var selectParts []string + var groupByParts []string + var args []interface{} + + // 基础审计字段（聚合模式下用常量） + selectParts = append(selectParts, "ROW_NUMBER() OVER () AS id") // 伪自增ID + selectParts = append(selectParts, "1 AS tenant_id") + selectParts = append(selectParts, fmt.Sprintf("'%s' AS business_code", ec.BusinessCode)) + selectParts = append(selectParts, "'system' AS creator") + selectParts = append(selectParts, "NOW() AS created_at") + selectParts = append(selectParts, "'system' AS updater") + selectParts = append(selectParts, "NOW() AS updated_at") + selectParts = append(selectParts, "NULL::TIMESTAMP AS deleted_at") + + // 日期字段（常量） + selectParts = append(selectParts, fmt.Sprintf("'%s' AS %s", statDate, dateField)) + + // 原始数据 + selectParts = append(selectParts, "'{}'::JSONB AS raw_data") + + // GroupByFields 集合（快速查找） + gbySet := make(map[string]bool) + for _, gbf := range ec.GroupByFields { + gbySet[gbf] = true + } + + // 添加 GroupBy 字段到 SELECT 和 GROUP BY + for _, gbf := range ec.GroupByFields { + selectParts = append(selectParts, fmt.Sprintf("%s.%s", sourceTable, gbf)) + groupByParts = append(groupByParts, fmt.Sprintf("%s.%s", sourceTable, gbf)) + } + + // 字段映射：根据 AggregateFunction 决定聚合方式 + for _, mapping := range ec.FieldMappings { + targetField := mapping.TargetField + sourceField := mapping.SourceField + + // 构建源表达式 + var sourceExpr string + if mapping.TransformRule != nil { + sourceExpr = e.applyTransformExpr(mapping.TransformRule, fmt.Sprintf("%s.%s", sourceTable, sourceField)) + } else { + sourceExpr = fmt.Sprintf("%s.%s", sourceTable, sourceField) + } + + // 判断是否需要聚合 + aggFunc := strings.ToUpper(mapping.AggregateFunction) + if aggFunc != "" && !gbySet[sourceField] { + // 聚合字段：SUM(s.xxx) / COUNT(s.xxx) / AVG(s.xxx) + expr := fmt.Sprintf("%s(%s)", aggFunc, sourceExpr) + if mapping.DefaultValue != nil { + expr = fmt.Sprintf("COALESCE(%s, %v)", expr, mapping.DefaultValue) + } + selectParts = append(selectParts, fmt.Sprintf("%s AS %s", expr, targetField)) + } else if gbySet[sourceField] { + // GroupBy 字段不需要重复加入 SELECT（已通过 groupByFields 处理） + continue + } else { + // 非聚合字段，也未在 GroupBy 中 → 用 MAX/MIN 取值（兼容 PG only_full_group_by） + expr := fmt.Sprintf("MAX(%s)", sourceExpr) + if mapping.DefaultValue != nil { + expr = fmt.Sprintf("COALESCE(%s, %v)", expr, mapping.DefaultValue) + } + selectParts = append(selectParts, fmt.Sprintf("%s AS %s", expr, targetField)) + } + } + + // FROM + JOIN + fromClause := e.buildFromClause(ec, sourceTable) + + // WHERE + whereClause := e.buildWhereClause(ec, sourceTable, statDate) + + // 组合 SQL + sql := fmt.Sprintf("SELECT %s FROM %s %s", + strings.Join(selectParts, ", "), + fromClause, + whereClause) + + // GROUP BY + if len(groupByParts) > 0 { + sql += " GROUP BY " + strings.Join(groupByParts, ", ") + } + + return sql, args, nil +} + +// buildFromClause 构建FROM + JOIN子句 +func (e *DailyExtractor) buildFromClause(ec *model.ExtractConfig, sourceTable string) string { + fromClause := fmt.Sprintf("%s %s", ec.SourceTableName, sourceTable) + + for _, join := range ec.JoinConfigs { + joinType := "LEFT JOIN" + jType := strings.ToUpper(join.JoinType) + if jType == "INNER" { + joinType = "INNER JOIN" + } else if jType == "RIGHT" { + joinType = "RIGHT JOIN" + } + + joinAlias := join.JoinAlias + if joinAlias == "" { + joinAlias = join.JoinTable + } + + fromClause += fmt.Sprintf(" %s %s %s ON %s", joinType, join.JoinTable, joinAlias, join.JoinCondition) + } + + return fromClause +} + +// buildJoinFieldSelects 构建JOIN表的字段映射SELECT部分 +func (e *DailyExtractor) buildJoinFieldSelects(ec *model.ExtractConfig) []string { + var parts []string + for _, join := range ec.JoinConfigs { + joinAlias := join.JoinAlias + if joinAlias == "" { + joinAlias = join.JoinTable + } + for _, jm := range join.FieldMappings { + targetField := jm.TargetField + sourceExpr := fmt.Sprintf("%s.%s", joinAlias, jm.SourceField) + if jm.TransformRule != nil { + sourceExpr = e.applyTransformExpr(jm.TransformRule, sourceExpr) + } + parts = append(parts, fmt.Sprintf("%s AS %s", sourceExpr, targetField)) + } + } + return parts +} + +// buildWhereClause 构建WHERE子句 +func (e *DailyExtractor) buildWhereClause(ec *model.ExtractConfig, sourceTable, statDate string) string { + var whereConditions []string + + // 日期范围（增量抽取） + if ec.ExtractType == model.ExtractTypeIncremental && ec.ExtractKeyField != "" { + dateCondition := fmt.Sprintf("%s.%s::date = '%s'", sourceTable, ec.ExtractKeyField, statDate) + whereConditions = append(whereConditions, dateCondition) + } + + // 自定义过滤条件 + if ec.FilterExpression != "" { + whereConditions = append(whereConditions, ec.FilterExpression) + } + + if len(whereConditions) == 0 { + return "" + } + return "WHERE " + strings.Join(whereConditions, " AND ") +} + +// applyTransformExpr 应用转换表达式 +func (e *DailyExtractor) applyTransformExpr(rule *model.TransformRule, sourceExpr string) string { + switch rule.RuleType { + case "CALCULATE": + if rule.Expression != "" { + return strings.ReplaceAll(rule.Expression, "{source}", sourceExpr) + } + case "FORMAT": + if rule.Format != "" { + return fmt.Sprintf("TO_CHAR(%s, '%s')", sourceExpr, rule.Format) + } + case "MAPPING": + // 在代码中运行时做映射 + return sourceExpr + } + return sourceExpr +} + +// applyTransformRules 应用运行时转换规则（映射等代码转换） +func (e *DailyExtractor) applyTransformRules(ec *model.ExtractConfig, row map[string]interface{}) { + for _, rule := range ec.TransformRules { + if rule.RuleType != "MAPPING" { + continue + } + + sourceField := rule.Expression // 存储源字段名 + targetField := rule.RuleCode // 存储目标字段名 + + if sourceVal, ok := row[sourceField]; ok { + strVal := gconv.String(sourceVal) + if mapped, exists := rule.Mapping[strVal]; exists { + row[targetField] = mapped + } + } + } +} + +// ensureStatTableExists 确保统计宽表存在 +func (e *DailyExtractor) ensureStatTableExists(ctx context.Context, report *model.ReportConfig, fieldMap map[string]*model.FieldConfig) error { + tableName := report.StatTableName + + // 检查表是否存在 + result, err := gfdb.DB(ctx).GetAll(ctx, "SELECT COUNT(*) FROM pg_tables WHERE tablename = $1", strings.ToLower(tableName)) + if err != nil { + return err + } + count := 0 + if len(result) > 0 { + count = result[0]["count"].Int() + } + + if count == 0 { + // 需要建表 + return e.createStatTable(ctx, report, fieldMap) + } + + logrus.Infof("统计宽表 %s 已存在", tableName) + return nil +} + +// createStatTable 创建统计宽表 +func (e *DailyExtractor) createStatTable(ctx context.Context, report *model.ReportConfig, fieldMap map[string]*model.FieldConfig) error { + var cols []string + + // 标准审计字段 + cols = append(cols, "id BIGSERIAL PRIMARY KEY") + cols = append(cols, "tenant_id BIGINT NOT NULL DEFAULT 0") + cols = append(cols, "business_code VARCHAR(64) NOT NULL DEFAULT ''") + cols = append(cols, "creator VARCHAR(64) DEFAULT ''") + cols = append(cols, "created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()") + cols = append(cols, "updater VARCHAR(64) DEFAULT ''") + cols = append(cols, "updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()") + cols = append(cols, "deleted_at TIMESTAMP WITH TIME ZONE") + + // 日期字段 + dateField := report.DateField + if dateField == "" { + dateField = "stat_date" + } + cols = append(cols, fmt.Sprintf("%s VARCHAR(16) NOT NULL DEFAULT ''", dateField)) + + // 业务字段 + for _, fc := range fieldMap { + fc := fc + colType := fieldTypeToPG(fc.FieldType) + cols = append(cols, fmt.Sprintf("%s %s", fc.FieldCode, colType)) + } + + // 原始数据 + cols = append(cols, "raw_data JSONB DEFAULT '{}'") + + tableName := report.StatTableName + sql := fmt.Sprintf("CREATE TABLE IF NOT EXISTS %s (\n %s\n)", tableName, strings.Join(cols, ",\n ")) + + logrus.Infof("创建统计宽表: %s", tableName) + + if _, err := gfdb.DB(ctx).Exec(ctx, sql); err != nil { + return fmt.Errorf("建表失败: %w", err) + } + + // 冲突唯一索引 + if len(report.ConflictKeys) > 0 { + indexName := fmt.Sprintf("uq_%s_conflict", tableName) + indexCols := strings.Join(report.ConflictKeys, ", ") + indexSQL := fmt.Sprintf("CREATE UNIQUE INDEX IF NOT EXISTS %s ON %s (%s)", indexName, tableName, indexCols) + if _, err := gfdb.DB(ctx).Exec(ctx, indexSQL); err != nil { + logrus.Warnf("创建冲突索引失败: %v", err) + } + } + + // 字段注释 + for _, fc := range fieldMap { + fc := fc + if fc.FieldName != "" { + escaped := strings.ReplaceAll(fc.FieldName, "'", "''") + commentSQL := fmt.Sprintf("COMMENT ON COLUMN %s.%s IS '%s'", tableName, fc.FieldCode, escaped) + if _, err := gfdb.DB(ctx).Exec(ctx, commentSQL); err != nil { + logrus.Warnf("添加字段注释失败 [%s.%s]: %v", tableName, fc.FieldCode, err) + } + } + } + + return nil +} + +// batchUpsert 批量upsert写入 +func (e *DailyExtractor) batchUpsert(ctx context.Context, tableName string, conflictKeys []string, rows []map[string]interface{}) (int, []string, error) { + if len(rows) == 0 { + return 0, nil, nil + } + + now := time.Now() + for i := range rows { + if rows[i] == nil { + rows[i] = make(map[string]interface{}) + } + rows[i]["updated_at"] = now + } + + batchSize := 100 + total := 0 + var allColumns []string + + for i := 0; i < len(rows); i += batchSize { + end := i + batchSize + if end > len(rows) { + end = len(rows) + } + batch := rows[i:end] + + m := gfdb.DB(ctx).Model(ctx, tableName).Data(batch) + if len(conflictKeys) > 0 { + keys := make([]interface{}, len(conflictKeys)) + for j, k := range conflictKeys { + keys[j] = k + } + m = m.OnConflict(keys...) + } + + _, err := m.Save() + if err != nil { + logrus.Errorf("批量写入 %s 失败: %v", tableName, err) + // 逐条重试 + for _, row := range batch { + mm := gfdb.DB(ctx).Model(ctx, tableName).Data(row) + if len(conflictKeys) > 0 { + keys := make([]interface{}, len(conflictKeys)) + for j, k := range conflictKeys { + keys[j] = k + } + mm = mm.OnConflict(keys...) + } + if _, e := mm.Save(); e != nil { + logrus.Errorf("逐条写入失败: %v", e) + } else { + total++ + } + } + } else { + total += len(batch) + } + } + + return total, allColumns, nil +} + +// fieldTypeToPG 字段类型转PG类型 +func fieldTypeToPG(fieldType string) string { + switch fieldType { + case model.FieldTypeInt: + return "NUMERIC(20,0) DEFAULT 0" + case model.FieldTypeFloat: + return "NUMERIC(20,4) DEFAULT 0" + case model.FieldTypeDate: + return "VARCHAR(16) DEFAULT ''" + case model.FieldTypeDatetime: + return "TIMESTAMP WITH TIME ZONE" + case model.FieldTypeJsonb: + return "JSONB DEFAULT '{}'" + default: + return "VARCHAR(256) DEFAULT ''" + } +} diff --git a/common/report/model/model.go b/common/report/model/model.go new file mode 100644 index 0000000..f318666 --- /dev/null +++ b/common/report/model/model.go @@ -0,0 +1,424 @@ +package model + +import ( + "time" +) + +// ============================================================ +// 实体定义 +// ============================================================ + +// BusinessConfig 业务配置 +type BusinessConfig struct { + ID int64 `orm:"id" json:"id"` + TenantId uint64 `orm:"tenant_id" json:"tenant_id"` + BusinessCode string `orm:"business_code" json:"businessCode"` + BusinessName string `orm:"business_name" json:"businessName"` + Description string `orm:"description" json:"description"` + Status string `orm:"status" json:"status"` + Config map[string]interface{} `orm:"config" json:"config"` + Creator string `orm:"creator" json:"creator"` + CreatedAt *time.Time `orm:"created_at" json:"createdAt"` + Updater string `orm:"updater" json:"updater"` + UpdatedAt *time.Time `orm:"updated_at" json:"updatedAt"` + DeletedAt *time.Time `orm:"deleted_at" json:"deletedAt"` +} + +// ReportConfig 报表配置 +type ReportConfig struct { + ID int64 `orm:"id" json:"id"` + TenantId uint64 `orm:"tenant_id" json:"tenant_id"` + BusinessCode string `orm:"business_code" json:"businessCode"` + ReportCode string `orm:"report_code" json:"reportCode"` + ReportName string `orm:"report_name" json:"reportName"` + Description string `orm:"description" json:"description"` + Status string `orm:"status" json:"status"` + StatTableName string `orm:"stat_table_name" json:"statTableName"` + StatTableComment string `orm:"stat_table_comment" json:"statTableComment"` + DateField string `orm:"date_field" json:"dateField"` + PrimaryKeys []string `orm:"primary_keys" json:"primaryKeys"` + ConflictKeys []string `orm:"conflict_keys" json:"conflictKeys"` + Config map[string]interface{} `orm:"config" json:"config"` + Creator string `orm:"creator" json:"creator"` + CreatedAt *time.Time `orm:"created_at" json:"createdAt"` + Updater string `orm:"updater" json:"updater"` + UpdatedAt *time.Time `orm:"updated_at" json:"updatedAt"` + DeletedAt *time.Time `orm:"deleted_at" json:"deletedAt"` +} + +// FieldConfig 字段配置 +type FieldConfig struct { + ID int64 `orm:"id" json:"id"` + TenantId uint64 `orm:"tenant_id" json:"tenant_id"` + BusinessCode string `orm:"business_code" json:"businessCode"` + ReportCode string `orm:"report_code" json:"reportCode"` + FieldCode string `orm:"field_code" json:"fieldCode"` + FieldName string `orm:"field_name" json:"fieldName"` + FieldType string `orm:"field_type" json:"fieldType"` + DataType string `orm:"data_type" json:"dataType"` + FieldRole string `orm:"field_role" json:"fieldRole"` + IsAggregatable bool `orm:"is_aggregatable" json:"isAggregatable"` + IsFilterable bool `orm:"is_filterable" json:"isFilterable"` + IsQueryable bool `orm:"is_queryable" json:"isQueryable"` + IsSortable bool `orm:"is_sortable" json:"isSortable"` + DefaultAggregate string `orm:"default_aggregate" json:"defaultAggregate"` + ValidAggregates []string `orm:"valid_aggregates" json:"validAggregates"` + FilterOperators []string `orm:"filter_operators" json:"filterOperators"` + Expression string `orm:"expression" json:"expression"` + ExpressionType string `orm:"expression_type" json:"expressionType"` + FormatPattern string `orm:"format_pattern" json:"formatPattern"` + Unit string `orm:"unit" json:"unit"` + DictCode string `orm:"dict_code" json:"dictCode"` + SortOrder int `orm:"sort_order" json:"sortOrder"` + GroupName string `orm:"group_name" json:"groupName"` + Status string `orm:"status" json:"status"` + Creator string `orm:"creator" json:"creator"` + CreatedAt *time.Time `orm:"created_at" json:"createdAt"` + Updater string `orm:"updater" json:"updater"` + UpdatedAt *time.Time `orm:"updated_at" json:"updatedAt"` + DeletedAt *time.Time `orm:"deleted_at" json:"deletedAt"` +} + +// ExtractConfig 抽取配置 +type ExtractConfig struct { + ID int64 `orm:"id" json:"id"` + TenantId uint64 `orm:"tenant_id" json:"tenant_id"` + BusinessCode string `orm:"business_code" json:"businessCode"` + ReportCode string `orm:"report_code" json:"reportCode"` + ExtractCode string `orm:"extract_code" json:"extractCode"` + ExtractName string `orm:"extract_name" json:"extractName"` + SourceTableName string `orm:"source_table_name" json:"sourceTableName"` + SourceTableAlias string `orm:"source_table_alias" json:"sourceTableAlias"` + TargetTableName string `orm:"target_table_name" json:"targetTableName"` + IsEnabled bool `orm:"is_enabled" json:"isEnabled"` + ExtractType string `orm:"extract_type" json:"extractType"` + ExtractMode string `orm:"extract_mode" json:"extractMode"` + ExtractKeyField string `orm:"extract_key_field" json:"extractKeyField"` + ExtractKeyFormat string `orm:"extract_key_format" json:"extractKeyFormat"` + GroupByFields []string `orm:"group_by_fields" json:"groupByFields"` + FilterExpression string `orm:"filter_expression" json:"filterExpression"` + JoinConfigs []JoinConfig `orm:"join_configs" json:"joinConfigs"` + FieldMappings []FieldMapping `orm:"field_mappings" json:"fieldMappings"` + TransformRules []TransformRule `orm:"transform_rules" json:"transformRules"` + BatchSize int `orm:"batch_size" json:"batchSize"` + Status string `orm:"status" json:"status"` + Creator string `orm:"creator" json:"creator"` + CreatedAt *time.Time `orm:"created_at" json:"createdAt"` + Updater string `orm:"updater" json:"updater"` + UpdatedAt *time.Time `orm:"updated_at" json:"updatedAt"` + DeletedAt *time.Time `orm:"deleted_at" json:"deletedAt"` +} + +// ExtractLog 抽取记录 +type ExtractLog struct { + ID int64 `orm:"id" json:"id"` + BusinessCode string `orm:"business_code" json:"businessCode"` + ReportCode string `orm:"report_code" json:"reportCode"` + ExtractCode string `orm:"extract_code" json:"extractCode"` + StatDate string `orm:"stat_date" json:"statDate"` + ExtractType string `orm:"extract_type" json:"extractType"` + Status string `orm:"status" json:"status"` + TotalCount int `orm:"total_count" json:"totalCount"` + SuccessCount int `orm:"success_count" json:"successCount"` + FailCount int `orm:"fail_count" json:"failCount"` + StartTime *time.Time `orm:"start_time" json:"startTime"` + EndTime *time.Time `orm:"end_time" json:"endTime"` + ErrorMessage string `orm:"error_message" json:"errorMessage"` + Executor string `orm:"executor" json:"executor"` + CreatedAt *time.Time `orm:"created_at" json:"createdAt"` + UpdatedAt *time.Time `orm:"updated_at" json:"updatedAt"` +} + +// ============================================================ +// 辅助结构 +// ============================================================ + +// JoinConfig 关联配置 +type JoinConfig struct { + JoinTable string `json:"joinTable"` + JoinAlias string `json:"joinAlias"` + JoinType string `json:"joinType"` // LEFT/RIGHT/INNER + JoinCondition string `json:"joinCondition"` + FieldMappings []FieldMapping `json:"fieldMappings"` +} + +// FieldMapping 字段映射 +type FieldMapping struct { + SourceField string `json:"sourceField"` + TargetField string `json:"targetField"` + FieldType string `json:"fieldType"` + AggregateFunction string `json:"aggregateFunction"` + DefaultValue interface{} `json:"defaultValue"` + TransformRule *TransformRule `json:"transformRule,omitempty"` +} + +// TransformRule 转换规则 +type TransformRule struct { + RuleCode string `json:"ruleCode"` + RuleType string `json:"ruleType"` // DIRECT/MAPPING/FORMAT/CALCULATE + Expression string `json:"expression"` + Format string `json:"format"` + Mapping map[string]interface{} `json:"mapping"` +} + +// ============================================================ +// 前端请求/响应结构体 +// ============================================================ + +// UserSelectQueryReq 用户选择查询请求 +type UserSelectQueryReq struct { + BusinessCode string `json:"businessCode" v:"required" dc:"业务编码"` + ReportCode string `json:"reportCode" v:"required" dc:"报表编码"` + Dimensions []string `json:"dimensions" dc:"统计维度列表，如 shop_id/anchor_id/date"` + Indicators []IndicatorSelect `json:"indicators" dc:"统计指标列表(含聚合方式)"` + Filters []FilterCondition `json:"filters" dc:"筛选条件列表"` + TimeRange *TimeRange `json:"timeRange" dc:"时间范围"` + TimeGroup string `json:"timeGroup" dc:"时间分组: day/week/month/quarter"` + OrderBy []OrderCondition `json:"orderBy" dc:"排序条件"` + Page int `json:"page" dc:"页码" d:"1"` + PageSize int `json:"pageSize" dc:"每页条数" d:"20"` +} + +// IndicatorSelect 指标选择 +type IndicatorSelect struct { + FieldCode string `json:"fieldCode" dc:"字段编码"` + Aggregate string `json:"aggregate" dc:"聚合方式: SUM/COUNT/AVG/MAX/MIN"` + Alias string `json:"alias" dc:"别名"` +} + +// FilterCondition 筛选条件 +type FilterCondition struct { + FieldCode string `json:"fieldCode" dc:"字段编码"` + Operator string `json:"operator" dc:"操作符: =/!=/>/=/<=/IN/LIKE/BETWEEN"` + Value interface{} `json:"value" dc:"值"` + Value2 interface{} `json:"value2" dc:"第二个值(BETWEEN时使用)"` +} + +// TimeRange 时间范围 +type TimeRange struct { + StartDate string `json:"startDate" dc:"开始日期 yyyy-MM-dd"` + EndDate string `json:"endDate" dc:"结束日期 yyyy-MM-dd"` +} + +// OrderCondition 排序条件 +type OrderCondition struct { + FieldCode string `json:"fieldCode" dc:"字段编码"` + Direction string `json:"direction" dc:"排序方向: ASC/DESC"` +} + +// UserSelectQueryResp 用户选择查询响应 +type UserSelectQueryResp struct { + List []map[string]interface{} `json:"list" dc:"数据列表"` + Total int64 `json:"total" dc:"总数"` + Page int `json:"page" dc:"当前页"` + PageSize int `json:"pageSize" dc:"每页条数"` + TotalPages int `json:"totalPages" dc:"总页数"` + Sql string `json:"sql,omitempty" dc:"执行的SQL(调试用)"` + ExecTimeMs int64 `json:"execTimeMs" dc:"执行耗时(毫秒)"` +} + +// ExtractDailyDataReq 按天抽取数据请求 +type ExtractDailyDataReq struct { + BusinessCode string `json:"businessCode" v:"required" dc:"业务编码"` + ReportCode string `json:"reportCode" v:"required" dc:"报表编码"` + StatDate string `json:"statDate" v:"required" dc:"统计日期 yyyy-MM-dd"` + Executor string `json:"executor" dc:"执行人"` +} + +// ExtractDailyDataResp 按天抽取数据响应 +type ExtractDailyDataResp struct { + Success bool `json:"success" dc:"是否成功"` + TotalCount int `json:"totalCount" dc:"总记录数"` + SuccessCount int `json:"successCount" dc:"成功记录数"` + FailCount int `json:"failCount" dc:"失败记录数"` + ExecTimeMs int64 `json:"execTimeMs" dc:"执行耗时(毫秒)"` + ErrorMsg string `json:"errorMsg" dc:"错误信息"` +} + +// AutoCreateStatTableReq 自动创建统计宽表请求 +type AutoCreateStatTableReq struct { + BusinessCode string `json:"businessCode" v:"required" dc:"业务编码"` + ReportCode string `json:"reportCode" v:"required" dc:"报表编码"` + Creator string `json:"creator" dc:"创建人"` +} + +// AutoCreateStatTableResp 自动创建统计宽表响应 +type AutoCreateStatTableResp struct { + Success bool `json:"success" dc:"是否成功"` + TableName string `json:"tableName" dc:"创建的表名"` + ColumnCount int `json:"columnCount" dc:"字段数量"` + ExecTimeMs int64 `json:"execTimeMs" dc:"执行耗时(毫秒)"` +} + +// GetReportFieldsResp 获取报表可用字段响应 +type GetReportFieldsResp struct { + BusinessCode string `json:"businessCode" dc:"业务编码"` + ReportCode string `json:"reportCode" dc:"报表编码"` + Dimensions []FieldConfig `json:"dimensions" dc:"维度字段列表"` + Indicators []FieldConfig `json:"indicators" dc:"指标字段列表"` + Filters []FieldConfig `json:"filters" dc:"筛选字段列表"` +} + +// ============================================================ +// 配置 CRUD 请求/响应 +// ============================================================ + +// SaveBusinessReq 保存业务配置请求（新增/修改合一） +type SaveBusinessReq struct { + ID *int64 `json:"id"` // 有值为更新，无值为新增 + BusinessCode string `json:"businessCode" v:"required" dc:"业务编码"` + BusinessName string `json:"businessName" v:"required" dc:"业务名称"` + Description string `json:"description" dc:"描述"` + Status string `json:"status" dc:"状态 ACTIVE/INACTIVE" d:"ACTIVE"` + Config map[string]interface{} `json:"config" dc:"扩展配置"` + Operator string `json:"operator" dc:"操作人"` +} + +// SaveReportReq 保存报表配置请求 +type SaveReportReq struct { + ID *int64 `json:"id"` + BusinessCode string `json:"businessCode" v:"required" dc:"业务编码"` + ReportCode string `json:"reportCode" v:"required" dc:"报表编码"` + ReportName string `json:"reportName" v:"required" dc:"报表名称"` + Description string `json:"description" dc:"描述"` + Status string `json:"status" dc:"状态" d:"ACTIVE"` + StatTableName string `json:"statTableName" v:"required" dc:"统计宽表名"` + StatTableComment string `json:"statTableComment" dc:"统计宽表注释"` + DateField string `json:"dateField" dc:"日期字段" d:"stat_date"` + PrimaryKeys []string `json:"primaryKeys" dc:"主键字段"` + ConflictKeys []string `json:"conflictKeys" dc:"冲突键（唯一索引）"` + Config map[string]interface{} `json:"config" dc:"扩展配置"` + Operator string `json:"operator" dc:"操作人"` +} + +// SaveFieldReq 保存字段配置请求 +type SaveFieldReq struct { + ID *int64 `json:"id"` + BusinessCode string `json:"businessCode" v:"required" dc:"业务编码"` + ReportCode string `json:"reportCode" v:"required" dc:"报表编码"` + FieldCode string `json:"fieldCode" v:"required" dc:"字段编码"` + FieldName string `json:"fieldName" v:"required" dc:"字段名称"` + FieldType string `json:"fieldType" v:"required" dc:"字段类型 STRING/INT/FLOAT/DATE/DATETIME/JSONB"` + DataType string `json:"dataType" dc:"数据存储类型" d:"STRING"` + FieldRole string `json:"fieldRole" v:"required" dc:"字段角色 DIMENSION/INDICATOR/FILTER/FILTER_ONLY"` + IsAggregatable bool `json:"isAggregatable" dc:"是否可聚合"` + IsFilterable bool `json:"isFilterable" dc:"是否可筛选" d:"true"` + IsQueryable bool `json:"isQueryable" dc:"是否可查询" d:"true"` + IsSortable bool `json:"isSortable" dc:"是否可排序" d:"true"` + DefaultAggregate string `json:"defaultAggregate" dc:"默认聚合方式"` + ValidAggregates []string `json:"validAggregates" dc:"可选聚合列表"` + FilterOperators []string `json:"filterOperators" dc:"可选操作符列表"` + Expression string `json:"expression" dc:"表达式（衍生字段）"` + ExpressionType string `json:"expressionType" dc:"表达式类型 DIRECT/CALCULATED"` + FormatPattern string `json:"formatPattern" dc:"格式化模板"` + Unit string `json:"unit" dc:"单位"` + DictCode string `json:"dictCode" dc:"字典编码"` + SortOrder int `json:"sortOrder" dc:"排序"` + GroupName string `json:"groupName" dc:"分组名称"` + Status string `json:"status" dc:"状态" d:"ACTIVE"` + Operator string `json:"operator" dc:"操作人"` +} + +// SaveExtractConfigReq 保存抽取配置请求 +type SaveExtractConfigReq struct { + ID *int64 `json:"id"` + BusinessCode string `json:"businessCode" v:"required" dc:"业务编码"` + ReportCode string `json:"reportCode" v:"required" dc:"报表编码"` + ExtractCode string `json:"extractCode" v:"required" dc:"抽取编码"` + ExtractName string `json:"extractName" v:"required" dc:"抽取名称"` + SourceTableName string `json:"sourceTableName" v:"required" dc:"源表名"` + SourceTableAlias string `json:"sourceTableAlias" dc:"源表别名"` + TargetTableName string `json:"targetTableName" v:"required" dc:"目标表名"` + IsEnabled bool `json:"isEnabled" dc:"是否启用" d:"true"` + ExtractType string `json:"extractType" dc:"抽取类型 FULL/INCREMENTAL" d:"INCREMENTAL"` + ExtractMode string `json:"extractMode" dc:"抽取模式 DIRECT/AGGREGATE" d:"DIRECT"` + ExtractKeyField string `json:"extractKeyField" dc:"抽取关键字段（增量依据）"` + ExtractKeyFormat string `json:"extractKeyFormat" dc:"关键字段格式"` + GroupByFields []string `json:"groupByFields" dc:"GROUP BY 字段列表"` + FilterExpression string `json:"filterExpression" dc:"过滤表达式"` + JoinConfigs []JoinConfig `json:"joinConfigs" dc:"JOIN配置"` + FieldMappings []FieldMapping `json:"fieldMappings" dc:"字段映射列表"` + TransformRules []TransformRule `json:"transformRules" dc:"转换规则列表"` + BatchSize int `json:"batchSize" dc:"批处理大小" d:"1000"` + Status string `json:"status" dc:"状态" d:"ACTIVE"` + Operator string `json:"operator" dc:"操作人"` +} + +// IdReq 通用 ID 请求 +type IdReq struct { + ID int64 `json:"id" v:"required" dc:"主键ID"` +} + +// SaveResult 写操作通用返回 +type SaveResult struct { + Success bool `json:"success"` + ID int64 `json:"id"` + Message string `json:"message"` +} + +// DeleteResult 删除结果 +type DeleteResult struct { + Success bool `json:"success"` + Message string `json:"message"` +} + +// GetExtractConfigsReq 获取抽取配置列表请求 +type GetExtractConfigsReq struct { + BusinessCode string `json:"businessCode" v:"required" dc:"业务编码"` + ReportCode string `json:"reportCode" v:"required" dc:"报表编码"` +} + +// ============================================================ +// 常量定义 +// ============================================================ + +const ( + // 状态 + StatusActive = "ACTIVE" + StatusInactive = "INACTIVE" + + // 字段角色 + RoleDimension = "DIMENSION" + RoleIndicator = "INDICATOR" + RoleFilter = "FILTER" + RoleFilterOnly = "FILTER_ONLY" + + // 字段类型 + FieldTypeString = "STRING" + FieldTypeInt = "INT" + FieldTypeFloat = "FLOAT" + FieldTypeDate = "DATE" + FieldTypeDatetime = "DATETIME" + FieldTypeJsonb = "JSONB" + + // 聚合方式 + AggregateSum = "SUM" + AggregateCount = "COUNT" + AggregateAvg = "AVG" + AggregateMax = "MAX" + AggregateMin = "MIN" + + // 操作符 + OperatorEq = "=" + OperatorNe = "!=" + OperatorGt = ">" + OperatorLt = "<" + OperatorGe = ">=" + OperatorLe = "<=" + OperatorIn = "IN" + OperatorLike = "LIKE" + OperatorBetween = "BETWEEN" + + // 抽取类型 + ExtractTypeFull = "FULL" + ExtractTypeIncremental = "INCREMENTAL" + + // 抽取模式 + ExtractModeDirect = "DIRECT" // 逐行抽取（默认，源表每行 → 宽表一行） + ExtractModeAggregate = "AGGREGATE" // 聚合抽取（按 GROUP BY 聚合，SUM/COUNT/AVG） + + // 抽取状态 + ExtractStatusRunning = "RUNNING" + ExtractStatusSuccess = "SUCCESS" + ExtractStatusFailed = "FAILED" +) diff --git a/common/report/report.go b/common/report/report.go new file mode 100644 index 0000000..a6d16a2 --- /dev/null +++ b/common/report/report.go @@ -0,0 +1,146 @@ +package report + +import ( + "context" + + "gitea.redpowerfuture.com/red-future/common/db/gfdb" +) + +// initTables 初始化系统表 +func initTables(ctx context.Context) error { + ddls := []string{ + // 业务配置表 + `CREATE TABLE IF NOT EXISTS report_business_config ( + id BIGSERIAL PRIMARY KEY, + tenant_id BIGINT NOT NULL DEFAULT 0, + business_code VARCHAR(64) NOT NULL, + business_name VARCHAR(128) NOT NULL, + description TEXT DEFAULT '', + status VARCHAR(16) NOT NULL DEFAULT 'ACTIVE', + config JSONB DEFAULT '{}', + creator VARCHAR(64) DEFAULT '', + created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(), + updater VARCHAR(64) DEFAULT '', + updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(), + deleted_at TIMESTAMP WITH TIME ZONE + )`, + + // 报表配置表 + `CREATE TABLE IF NOT EXISTS report_report_config ( + id BIGSERIAL PRIMARY KEY, + tenant_id BIGINT NOT NULL DEFAULT 0, + business_code VARCHAR(64) NOT NULL, + report_code VARCHAR(64) NOT NULL, + report_name VARCHAR(128) NOT NULL, + description TEXT DEFAULT '', + status VARCHAR(16) NOT NULL DEFAULT 'ACTIVE', + stat_table_name VARCHAR(128) NOT NULL, + stat_table_comment VARCHAR(256) DEFAULT '', + date_field VARCHAR(64) DEFAULT 'stat_date', + primary_keys JSONB DEFAULT '["id"]'::jsonb, + conflict_keys JSONB DEFAULT '["stat_date"]'::jsonb, + config JSONB DEFAULT '{}', + creator VARCHAR(64) DEFAULT '', + created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(), + updater VARCHAR(64) DEFAULT '', + updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(), + deleted_at TIMESTAMP WITH TIME ZONE, + CONSTRAINT uk_business_report_code UNIQUE (tenant_id, business_code, report_code) + )`, + + // 字段配置表 + `CREATE TABLE IF NOT EXISTS report_field_config ( + id BIGSERIAL PRIMARY KEY, + tenant_id BIGINT NOT NULL DEFAULT 0, + business_code VARCHAR(64) NOT NULL, + report_code VARCHAR(64) NOT NULL, + field_code VARCHAR(64) NOT NULL, + field_name VARCHAR(128) NOT NULL, + field_type VARCHAR(32) NOT NULL, + data_type VARCHAR(32) NOT NULL DEFAULT 'STRING', + field_role VARCHAR(32) NOT NULL, + is_aggregatable BOOLEAN DEFAULT FALSE, + is_filterable BOOLEAN DEFAULT TRUE, + is_queryable BOOLEAN DEFAULT TRUE, + is_sortable BOOLEAN DEFAULT TRUE, + default_aggregate VARCHAR(32) DEFAULT '', + valid_aggregates JSONB DEFAULT '[]'::jsonb, + filter_operators JSONB DEFAULT '["=","!=",">","<",">=","<=","IN","LIKE","BETWEEN"]'::jsonb, + expression VARCHAR(512) DEFAULT '', + expression_type VARCHAR(32) DEFAULT '', + format_pattern VARCHAR(64) DEFAULT '', + unit VARCHAR(32) DEFAULT '', + dict_code VARCHAR(64) DEFAULT '', + sort_order INT DEFAULT 0, + group_name VARCHAR(64) DEFAULT '', + status VARCHAR(16) NOT NULL DEFAULT 'ACTIVE', + creator VARCHAR(64) DEFAULT '', + created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(), + updater VARCHAR(64) DEFAULT '', + updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(), + deleted_at TIMESTAMP WITH TIME ZONE, + CONSTRAINT uk_business_report_field_code UNIQUE (tenant_id, business_code, report_code, field_code) + )`, + + // 抽取配置表 + `CREATE TABLE IF NOT EXISTS report_extract_config ( + id BIGSERIAL PRIMARY KEY, + tenant_id BIGINT NOT NULL DEFAULT 0, + business_code VARCHAR(64) NOT NULL, + report_code VARCHAR(64) NOT NULL, + extract_code VARCHAR(64) NOT NULL, + extract_name VARCHAR(128) NOT NULL, + source_table_name VARCHAR(128) NOT NULL, + source_table_alias VARCHAR(64) DEFAULT '', + target_table_name VARCHAR(128) NOT NULL, + is_enabled BOOLEAN DEFAULT TRUE, + extract_type VARCHAR(32) NOT NULL DEFAULT 'FULL', + extract_mode VARCHAR(32) NOT NULL DEFAULT 'DIRECT', + extract_key_field VARCHAR(64) DEFAULT '', + extract_key_format VARCHAR(64) DEFAULT '', + group_by_fields JSONB DEFAULT '[]'::jsonb, + filter_expression TEXT DEFAULT '', + join_configs JSONB DEFAULT '[]'::jsonb, + field_mappings JSONB DEFAULT '[]'::jsonb, + transform_rules JSONB DEFAULT '[]'::jsonb, + batch_size INT DEFAULT 1000, + status VARCHAR(16) NOT NULL DEFAULT 'ACTIVE', + creator VARCHAR(64) DEFAULT '', + created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(), + updater VARCHAR(64) DEFAULT '', + updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(), + deleted_at TIMESTAMP WITH TIME ZONE, + CONSTRAINT uk_business_report_extract_code UNIQUE (tenant_id, business_code, report_code, extract_code) + )`, + + // 抽取记录表 + `CREATE TABLE IF NOT EXISTS report_extract_log ( + id BIGSERIAL PRIMARY KEY, + tenant_id BIGINT NOT NULL DEFAULT 0, + business_code VARCHAR(64) NOT NULL, + report_code VARCHAR(64) NOT NULL, + extract_code VARCHAR(64) NOT NULL, + stat_date VARCHAR(16) NOT NULL, + extract_type VARCHAR(32) NOT NULL, + status VARCHAR(16) NOT NULL DEFAULT 'RUNNING', + total_count INT DEFAULT 0, + success_count INT DEFAULT 0, + fail_count INT DEFAULT 0, + start_time TIMESTAMP WITH TIME ZONE DEFAULT NOW(), + end_time TIMESTAMP WITH TIME ZONE, + error_message TEXT DEFAULT '', + executor VARCHAR(64) DEFAULT '', + created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(), + updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(), + CONSTRAINT uk_extract_keys UNIQUE (tenant_id, business_code, report_code, extract_code, stat_date) + )`, + } + + for _, ddl := range ddls { + if _, err := gfdb.DB(ctx).Exec(ctx, ddl); err != nil { + return err + } + } + + return nil +} diff --git a/config.yml b/config.yml index 131a189..1e882ea 100644 --- a/config.yml +++ b/config.yml @@ -17,6 +17,7 @@ sync: sync_interval_minutes: 60 # 自动同步间隔（分钟） compensation_interval_seconds: 300 # 补偿调度器扫描间隔（秒） auto_sync_enabled: false # 是否启用自动同步 + sync_timeout_minutes: 120 # 单次同步超时（分钟），全量超大表可适当调大 # Database. database: diff --git a/consts/api-feature/application_status.go b/consts/api-feature/application_status.go index 250449d..4a13dec 100644 --- a/consts/api-feature/application_status.go +++ b/consts/api-feature/application_status.go @@ -1,13 +1,10 @@ package api_feature -// AppStatus 应用状态 -type AppStatus string +// ApplicationStatus 应用状态（暂未使用，保留供未来扩展） +type ApplicationStatus string const ( - AppStatusActive AppStatus = "active" // 启用 - AppStatusInactive AppStatus = "inactive" // 停用 + ApplicationStatusPending ApplicationStatus = "pending" + ApplicationStatusApproved ApplicationStatus = "approved" + ApplicationStatusRejected ApplicationStatus = "rejected" ) - -func (s AppStatus) String() string { - return string(s) -} diff --git a/consts/api-feature/application_type.go b/consts/api-feature/application_type.go index 0bbc471..2fd4d10 100644 --- a/consts/api-feature/application_type.go +++ b/consts/api-feature/application_type.go @@ -1,17 +1,10 @@ package api_feature -// AppType 应用类型 -type AppType string +// ApplicationType 应用类型（暂未使用，保留供未来扩展） +type ApplicationType string const ( - AppTypeWeb AppType = "web" // Web应用 - AppTypeMobile AppType = "mobile" // 移动应用 - AppTypeMiniApp AppType = "mini_app" // 小程序 - AppTypeH5 AppType = "h5" // H5应用 - AppTypeDesktop AppType = "desktop" // 桌面应用 - AppTypeThirdParty AppType = "third_party" // 第三方应用 + ApplicationTypeAPI ApplicationType = "api" + ApplicationTypeWebhook ApplicationType = "webhook" + ApplicationTypeSDK ApplicationType = "sdk" ) - -func (s AppType) String() string { - return string(s) -} diff --git a/consts/api-feature/fetch_status.go b/consts/api-feature/fetch_status.go index 232db9f..68b38a5 100644 --- a/consts/api-feature/fetch_status.go +++ b/consts/api-feature/fetch_status.go @@ -1,16 +1,11 @@ package api_feature -// FetchStatus 数据获取状态 +// FetchStatus 数据拉取状态（暂未使用，保留供未来扩展） type FetchStatus string const ( - FetchStatusPending FetchStatus = "pending" // 待执行 - FetchStatusRunning FetchStatus = "running" // 执行中 - FetchStatusSuccess FetchStatus = "success" // 成功 - FetchStatusFailed FetchStatus = "failed" // 失败 - FetchStatusRateLimit FetchStatus = "rate_limit" // 触发限流 + FetchStatusPending FetchStatus = "pending" + FetchStatusRunning FetchStatus = "running" + FetchStatusCompleted FetchStatus = "completed" + FetchStatusFailed FetchStatus = "failed" ) - -func (f FetchStatus) String() string { - return string(f) -} diff --git a/consts/api-feature/limit_type.go b/consts/api-feature/limit_type.go index 5e93f69..e861841 100644 --- a/consts/api-feature/limit_type.go +++ b/consts/api-feature/limit_type.go @@ -1,14 +1,9 @@ package api_feature -// LimitType 限流类型 +// LimitType 限流类型（暂未使用，保留供未来扩展） type LimitType string const ( - LimitTypeApp LimitType = "app" // 应用维度限流 - LimitTypeTenant LimitType = "tenant" // 租户维度限流 - LimitTypeApi LimitType = "api" // 接口维度限流 + LimitTypeRate LimitType = "rate" + LimitTypeBurst LimitType = "burst" ) - -func (l LimitType) String() string { - return string(l) -} diff --git a/consts/api-feature/mapping_status.go b/consts/api-feature/mapping_status.go index 41adf3d..730a25a 100644 --- a/consts/api-feature/mapping_status.go +++ b/consts/api-feature/mapping_status.go @@ -1,13 +1,9 @@ package api_feature -// MappingStatus 映射状态 +// MappingStatus 字段映射状态（暂未使用，保留供未来扩展） type MappingStatus string const ( - MappingStatusActive MappingStatus = "active" // 启用 - MappingStatusInactive MappingStatus = "inactive" // 停用 + MappingStatusMapped MappingStatus = "mapped" + MappingStatusUnmapped MappingStatus = "unmapped" ) - -func (s MappingStatus) String() string { - return string(s) -} diff --git a/consts/api-feature/platform_type.go b/consts/api-feature/platform_type.go index d443fe4..9f3827d 100644 --- a/consts/api-feature/platform_type.go +++ b/consts/api-feature/platform_type.go @@ -1,21 +1,11 @@ package api_feature -// SyncPlatform 同步平台类型 -type SyncPlatform string +// PlatformType 平台类型（暂未使用，保留供未来扩展） +type PlatformType string const ( - PlatformTaobao SyncPlatform = "taobao" // 淘宝 - PlatformJD SyncPlatform = "jd" // 京东 - PlatformKuaishou SyncPlatform = "kuaishou" // 快手 - PlatformDouyin SyncPlatform = "douyin" // 抖音 - PlatformXhs SyncPlatform = "xhs" // 小红书 - PlatformPdd SyncPlatform = "pdd" // 拼多多 - PlatformXianyu SyncPlatform = "xianyu" // 闲鱼 - PlatformTmall SyncPlatform = "tmall" // 天猫 - PlatformWechat SyncPlatform = "wechat" // 微信 - PlatformCustom SyncPlatform = "custom" // 自定义平台 + PlatformTypeAds PlatformType = "ads" + PlatformTypeEcom PlatformType = "ecom" + PlatformTypeSocial PlatformType = "social" + PlatformTypeDingTalk PlatformType = "dingtalk" ) - -func (s SyncPlatform) String() string { - return string(s) -} diff --git a/consts/api-feature/transform_type.go b/consts/api-feature/transform_type.go index 3154e83..f25aa2f 100644 --- a/consts/api-feature/transform_type.go +++ b/consts/api-feature/transform_type.go @@ -1,16 +1,13 @@ package api_feature -// TransformType 转换类型 +// TransformType 字段转换类型（暂未使用，保留供未来扩展） type TransformType string const ( - TransformTypeFixed TransformType = "fixed" // 固定值 - TransformTypeMapping TransformType = "mapping" // 值映射 - TransformTypeRegex TransformType = "regex" // 正则转换 - TransformTypeFunction TransformType = "function" // 函数转换 - TransformTypeScript TransformType = "script" // 脚本转换 + TransformTypeDirect TransformType = "direct" + TransformTypeFormatDate TransformType = "formatDate" + TransformTypeMapValue TransformType = "mapValue" + TransformTypeConcat TransformType = "concat" + TransformTypeCalc TransformType = "calc" + TransformTypeRegex TransformType = "regex" ) - -func (t TransformType) String() string { - return string(t) -} diff --git a/consts/dict/consts.go b/consts/dict/consts.go index 1f34334..eb27ef4 100644 --- a/consts/dict/consts.go +++ b/consts/dict/consts.go @@ -1,6 +1,12 @@ package dict -// 转换类型 +// 注意：请不要在此文件添加与 consts/api-feature 包重复的类型定义。 +// PlatformStatus 和 ApiMethod 等类型已在 dataengine/consts/api-feature 中定义， +// 请使用 api_feature.PlatformStatus / api_feature.ApiMethod 替代。 + +// Package dict 仅提供该包特有时使用。 + +// 转换类型（暂未使用，保留供未来扩展） const ( TransformTypeDirect = "direct" // 直接映射 TransformTypeFormatDate = "formatDate" // 日期格式化 @@ -10,7 +16,7 @@ const ( TransformTypeRegex = "regex" // 正则提取 ) -// 业务域 +// 业务域（暂未使用，保留供未来扩展） const ( BusinessDomainUser = "user" // 用户 BusinessDomainOrder = "order" // 订单 @@ -18,7 +24,7 @@ const ( BusinessDomainPayment = "payment" // 支付 ) -// 字段类型 +// 字段类型（暂未使用，保留供未来扩展） const ( FieldTypeString = "string" FieldTypeNumber = "number" @@ -30,22 +36,3 @@ const ( FieldTypeTime = "time" FieldTypeDateTime = "datetime" ) - -// 平台状态 -type PlatformStatus string - -const ( - PlatformStatusActive PlatformStatus = "active" - PlatformStatusInactive PlatformStatus = "inactive" -) - -// 接口方法 -type ApiMethod string - -const ( - ApiMethodGET ApiMethod = "GET" - ApiMethodPOST ApiMethod = "POST" - ApiMethodPUT ApiMethod = "PUT" - ApiMethodDELETE ApiMethod = "DELETE" - ApiMethodPATCH ApiMethod = "PATCH" -) diff --git a/controller/public/public_query_controller.go b/controller/public/public_query_controller.go new file mode 100644 index 0000000..31fe020 --- /dev/null +++ b/controller/public/public_query_controller.go @@ -0,0 +1,41 @@ +package public + +import ( + "context" + + dto "dataengine/model/dto/public" + svc "dataengine/service/public" + + "github.com/gogf/gf/v2/frame/g" +) + +type publicQueryController struct{} + +// PublicQuery 公共查询控制器 +var PublicQuery = new(publicQueryController) + +// QueryReq 查询请求（包含空结构用于路径） +type ClearCacheReq struct { + g.Meta `path:"/public/cache/clear" method:"delete" tags:"公共查询" summary:"清除查询缓存"` +} + +// Query 执行公共查询 +func (c *publicQueryController) Query(ctx context.Context, req *dto.QueryReq) (res *dto.QueryRes, err error) { + return svc.PublicQuery.Query(ctx, req) +} + +// GetTableList 获取可查询表列表 +func (c *publicQueryController) GetTableList(ctx context.Context, req *dto.TableListReq) (res *dto.TableListRes, err error) { + return svc.PublicQuery.GetTableList(ctx) +} + +// GetColumnList 获取表字段列表 +func (c *publicQueryController) GetColumnList(ctx context.Context, req *dto.ColumnListReq) (res *dto.ColumnListRes, err error) { + return svc.PublicQuery.GetColumnList(ctx, req.Table) +} + +// ClearCache 清除查询缓存（管理接口） +func (c *publicQueryController) ClearCache(ctx context.Context, req *ClearCacheReq) (res *dto.TableListRes, err error) { + svc.PublicQuery.ClearTableCache() + return &dto.TableListRes{List: []dto.TableInfo{}}, nil +} diff --git a/controller/report/report_admin_controller.go b/controller/report/report_admin_controller.go new file mode 100644 index 0000000..ae880be --- /dev/null +++ b/controller/report/report_admin_controller.go @@ -0,0 +1,949 @@ +package report + +import ( + "github.com/gogf/gf/v2/net/ghttp" +) + +// ReportAdminPage 报表引擎管理页面 +func ReportAdminPage(r *ghttp.Request) { + r.Response.Header().Set("Content-Type", "text/html; charset=utf-8") + r.Response.Write(reportAdminHTML) + r.Exit() +} + +var reportAdminHTML = ` + + + + +报表引擎管理 + + + +
+

📊 报表引擎管理

+ ← 返回管理后台 +
+
+ +
+
业务管理
+
报表配置
+
字段配置
+
抽取配置
+
数据查询
+
+ + +
+
+

业务列表

+
+ + +
+
+
加载中...
+
+
+ + +
+
+
+

报表列表

+ +
+
+ +
+
+ +
+
+ + +
+
+
+

字段列表

+ + +
+
+ +
+
+ +
+
+ + +
+
+
+

抽取配置列表

+ + +
+
+ + +
+
+ +
+
+ + +
+
+
+

数据查询

+ + +
+
+ + +
+ +
+ + + + + + + +` diff --git a/controller/report/report_controller.go b/controller/report/report_controller.go new file mode 100644 index 0000000..7b0d451 --- /dev/null +++ b/controller/report/report_controller.go @@ -0,0 +1,371 @@ +package report + +import ( + "context" + + reportSvc "dataengine/common/report" + "dataengine/common/report/model" + + "gitea.redpowerfuture.com/red-future/common/beans" + "github.com/gogf/gf/v2/frame/g" +) + +type report struct{} + +var ReportController = new(report) + +func svc() *reportSvc.ReportService { + return reportSvc.GetService() +} + +func ctxWithUser(ctx context.Context) context.Context { + return context.WithValue(ctx, "user", &beans.User{UserName: "admin", TenantId: 1}) +} + +// ============================================================ +// 业务 CRUD +// ============================================================ + +type listBusinessesReq struct { + g.Meta `path:"/businesses" method:"get" tags:"报表引擎" summary:"业务列表"` +} + +type listBusinessesRes struct { + List []model.BusinessConfig `json:"list"` +} + +func (c *report) ListBusinesses(ctx context.Context, req *listBusinessesReq) (*listBusinessesRes, error) { + ctx = ctxWithUser(ctx) + list, err := svc().GetAllBusinesses(ctx) + if err != nil { + return nil, err + } + return &listBusinessesRes{List: list}, nil +} + +type getBusinessReq struct { + g.Meta `path:"/business" method:"get" tags:"报表引擎" summary:"获取业务"` + ID int64 `json:"id" v:"required"` +} + +type getBusinessRes struct { + Data *model.BusinessConfig `json:"data"` +} + +func (c *report) GetBusiness(ctx context.Context, req *getBusinessReq) (*getBusinessRes, error) { + ctx = ctxWithUser(ctx) + data, err := svc().GetBusiness(ctx, req.ID) + if err != nil { + return nil, err + } + return &getBusinessRes{Data: data}, nil +} + +type saveBusinessReq struct { + g.Meta `path:"/business/save" method:"post" tags:"报表引擎" summary:"保存业务"` + model.SaveBusinessReq +} + +type saveBusinessRes struct { + *model.SaveResult +} + +func (c *report) SaveBusiness(ctx context.Context, req *saveBusinessReq) (*saveBusinessRes, error) { + ctx = ctxWithUser(ctx) + result, err := svc().SaveBusiness(ctx, &req.SaveBusinessReq) + if err != nil { + return nil, err + } + return &saveBusinessRes{SaveResult: result}, nil +} + +type deleteBusinessReq struct { + g.Meta `path:"/business" method:"delete" tags:"报表引擎" summary:"删除业务"` + ID int64 `json:"id" v:"required"` +} + +type deleteBusinessRes struct { + *model.DeleteResult +} + +func (c *report) DeleteBusiness(ctx context.Context, req *deleteBusinessReq) (*deleteBusinessRes, error) { + ctx = ctxWithUser(ctx) + result, err := svc().DeleteBusiness(ctx, req.ID) + if err != nil { + return nil, err + } + return &deleteBusinessRes{DeleteResult: result}, nil +} + +// ============================================================ +// 报表 CRUD +// ============================================================ + +type listReportsReq struct { + g.Meta `path:"/reports" method:"get" tags:"报表引擎" summary:"报表列表"` + BusinessCode string `json:"businessCode" v:"required"` +} + +type listReportsRes struct { + List []model.ReportConfig `json:"list"` +} + +func (c *report) ListReports(ctx context.Context, req *listReportsReq) (*listReportsRes, error) { + ctx = ctxWithUser(ctx) + list, err := svc().GetAllReports(ctx, req.BusinessCode) + if err != nil { + return nil, err + } + return &listReportsRes{List: list}, nil +} + +type getReportReq struct { + g.Meta `path:"/report" method:"get" tags:"报表引擎" summary:"获取报表"` + ID int64 `json:"id" v:"required"` +} + +type getReportRes struct { + Data *model.ReportConfig `json:"data"` +} + +func (c *report) GetReport(ctx context.Context, req *getReportReq) (*getReportRes, error) { + ctx = ctxWithUser(ctx) + data, err := svc().GetReport(ctx, req.ID) + if err != nil { + return nil, err + } + return &getReportRes{Data: data}, nil +} + +type saveReportReq struct { + g.Meta `path:"/report/save" method:"post" tags:"报表引擎" summary:"保存报表"` + model.SaveReportReq +} + +type saveReportRes struct { + *model.SaveResult +} + +func (c *report) SaveReport(ctx context.Context, req *saveReportReq) (*saveReportRes, error) { + ctx = ctxWithUser(ctx) + result, err := svc().SaveReport(ctx, &req.SaveReportReq) + if err != nil { + return nil, err + } + return &saveReportRes{SaveResult: result}, nil +} + +type deleteReportReq struct { + g.Meta `path:"/report" method:"delete" tags:"报表引擎" summary:"删除报表"` + ID int64 `json:"id" v:"required"` +} + +type deleteReportRes struct { + *model.DeleteResult +} + +func (c *report) DeleteReport(ctx context.Context, req *deleteReportReq) (*deleteReportRes, error) { + ctx = ctxWithUser(ctx) + result, err := svc().DeleteReport(ctx, req.ID) + if err != nil { + return nil, err + } + return &deleteReportRes{DeleteResult: result}, nil +} + +// ============================================================ +// 字段 CRUD +// ============================================================ + +type getReportFieldsReq struct { + g.Meta `path:"/fields" method:"get" tags:"报表引擎" summary:"报表字段列表（按角色分组）"` + BusinessCode string `json:"businessCode" v:"required"` + ReportCode string `json:"reportCode" v:"required"` +} + +func (c *report) GetReportFields(ctx context.Context, req *getReportFieldsReq) (*model.GetReportFieldsResp, error) { + ctx = ctxWithUser(ctx) + return svc().GetReportFields(ctx, req.BusinessCode, req.ReportCode) +} + +type getFieldReq struct { + g.Meta `path:"/field" method:"get" tags:"报表引擎" summary:"获取字段"` + ID int64 `json:"id" v:"required"` +} + +type getFieldRes struct { + Data *model.FieldConfig `json:"data"` +} + +func (c *report) GetField(ctx context.Context, req *getFieldReq) (*getFieldRes, error) { + ctx = ctxWithUser(ctx) + data, err := svc().GetField(ctx, req.ID) + if err != nil { + return nil, err + } + return &getFieldRes{Data: data}, nil +} + +type saveFieldReq struct { + g.Meta `path:"/field/save" method:"post" tags:"报表引擎" summary:"保存字段"` + model.SaveFieldReq +} + +type saveFieldRes struct { + *model.SaveResult +} + +func (c *report) SaveField(ctx context.Context, req *saveFieldReq) (*saveFieldRes, error) { + ctx = ctxWithUser(ctx) + result, err := svc().SaveField(ctx, &req.SaveFieldReq) + if err != nil { + return nil, err + } + return &saveFieldRes{SaveResult: result}, nil +} + +type deleteFieldReq struct { + g.Meta `path:"/field" method:"delete" tags:"报表引擎" summary:"删除字段"` + ID int64 `json:"id" v:"required"` +} + +type deleteFieldRes struct { + *model.DeleteResult +} + +func (c *report) DeleteField(ctx context.Context, req *deleteFieldReq) (*deleteFieldRes, error) { + ctx = ctxWithUser(ctx) + result, err := svc().DeleteField(ctx, req.ID) + if err != nil { + return nil, err + } + return &deleteFieldRes{DeleteResult: result}, nil +} + +// ============================================================ +// 抽取配置 CRUD +// ============================================================ + +type getExtractConfigsReq struct { + g.Meta `path:"/extractConfigs" method:"get" tags:"报表引擎" summary:"抽取配置列表"` + BusinessCode string `json:"businessCode" v:"required"` + ReportCode string `json:"reportCode" v:"required"` +} + +type getExtractConfigsRes struct { + List []model.ExtractConfig `json:"list"` +} + +func (c *report) GetExtractConfigs(ctx context.Context, req *getExtractConfigsReq) (*getExtractConfigsRes, error) { + ctx = ctxWithUser(ctx) + list, err := svc().GetExtractConfigs(ctx, req.BusinessCode, req.ReportCode) + if err != nil { + return nil, err + } + return &getExtractConfigsRes{List: list}, nil +} + +type getExtractConfigReq struct { + g.Meta `path:"/extractConfig" method:"get" tags:"报表引擎" summary:"获取抽取配置"` + ID int64 `json:"id" v:"required"` +} + +type getExtractConfigRes struct { + Data *model.ExtractConfig `json:"data"` +} + +func (c *report) GetExtractConfig(ctx context.Context, req *getExtractConfigReq) (*getExtractConfigRes, error) { + ctx = ctxWithUser(ctx) + data, err := svc().GetExtractConfig(ctx, req.ID) + if err != nil { + return nil, err + } + return &getExtractConfigRes{Data: data}, nil +} + +type saveExtractConfigReq struct { + g.Meta `path:"/extractConfig/save" method:"post" tags:"报表引擎" summary:"保存抽取配置"` + model.SaveExtractConfigReq +} + +type saveExtractConfigRes struct { + *model.SaveResult +} + +func (c *report) SaveExtractConfig(ctx context.Context, req *saveExtractConfigReq) (*saveExtractConfigRes, error) { + ctx = ctxWithUser(ctx) + result, err := svc().SaveExtractConfig(ctx, &req.SaveExtractConfigReq) + if err != nil { + return nil, err + } + return &saveExtractConfigRes{SaveResult: result}, nil +} + +type deleteExtractConfigReq struct { + g.Meta `path:"/extractConfig" method:"delete" tags:"报表引擎" summary:"删除抽取配置"` + ID int64 `json:"id" v:"required"` +} + +type deleteExtractConfigRes struct { + *model.DeleteResult +} + +func (c *report) DeleteExtractConfig(ctx context.Context, req *deleteExtractConfigReq) (*deleteExtractConfigRes, error) { + ctx = ctxWithUser(ctx) + result, err := svc().DeleteExtractConfig(ctx, req.ID) + if err != nil { + return nil, err + } + return &deleteExtractConfigRes{DeleteResult: result}, nil +} + +// ============================================================ +// 数据操作 +// ============================================================ + +type extractDataReq struct { + g.Meta `path:"/extract" method:"post" tags:"报表引擎" summary:"执行按天数据抽取"` + model.ExtractDailyDataReq +} + +func (c *report) ExtractData(ctx context.Context, req *extractDataReq) (*model.ExtractDailyDataResp, error) { + ctx = ctxWithUser(ctx) + return svc().ExtractDailyData(ctx, req.BusinessCode, req.ReportCode, req.StatDate, req.Executor) +} + +type autoCreateTableReq struct { + g.Meta `path:"/autoCreateTable" method:"post" tags:"报表引擎" summary:"自动创建统计宽表"` + model.AutoCreateStatTableReq +} + +func (c *report) AutoCreateTable(ctx context.Context, req *autoCreateTableReq) (*model.AutoCreateStatTableResp, error) { + ctx = ctxWithUser(ctx) + return svc().AutoCreateStatTable(ctx, req.BusinessCode, req.ReportCode) +} + +type queryReportReq struct { + g.Meta `path:"/query" method:"post" tags:"报表引擎" summary:"用户选择查询"` + model.UserSelectQueryReq +} + +func (c *report) QueryReport(ctx context.Context, req *queryReportReq) (*model.UserSelectQueryResp, error) { + ctx = ctxWithUser(ctx) + return svc().QueryReportByUserSelect(ctx, &req.UserSelectQueryReq) +} + +type initTablesReq struct { + g.Meta `path:"/initTables" method:"post" tags:"报表引擎" summary:"初始化系统表"` +} + +type initTablesRes struct { + Success bool `json:"success"` + Message string `json:"message"` +} + +func (c *report) InitTables(ctx context.Context, req *initTablesReq) (*initTablesRes, error) { + ctx = ctxWithUser(ctx) + if err := svc().InitSystemTables(ctx); err != nil { + return nil, err + } + return &initTablesRes{Success: true, Message: "系统表初始化完成"}, nil +} diff --git a/dao/dict/api_datasource_platform_dao.go b/dao/dict/api_datasource_platform_dao.go index d14af0c..cb44cb5 100644 --- a/dao/dict/api_datasource_platform_dao.go +++ b/dao/dict/api_datasource_platform_dao.go @@ -2,11 +2,9 @@ package dict import ( "context" - consts1 "dataengine/consts/api-feature" consts "dataengine/consts/public" dto "dataengine/model/dto/dict" entity "dataengine/model/entity/dict" - "strconv" "time" "gitea.redpowerfuture.com/red-future/common/db/gfdb" @@ -174,7 +172,7 @@ func (d *datasourcePlatformDao) ExistsByPlatformCode(ctx context.Context, platfo // ListActivePlatforms 获取所有启用的平台 func (d *datasourcePlatformDao) ListActivePlatforms(ctx context.Context) (res []entity.DatasourcePlatform, err error) { r, err := gfdb.DB(ctx).Model(ctx, consts.DatasourcePlatformTable). - Where(entity.DatasourcePlatformCols.Status, consts1.PlatformStatusActive). + Where(entity.DatasourcePlatformCols.Status, "ACTIVE"). OrderAsc(entity.DatasourcePlatformCols.PlatformName). All() if err != nil { @@ -197,7 +195,7 @@ func (d *datasourcePlatformDao) GetPlatformStatistics(ctx context.Context) (stat // 启用平台数 active, err := gfdb.DB(ctx).Model(ctx, consts.DatasourcePlatformTable). - Where(entity.DatasourcePlatformCols.Status, consts1.MappingStatusActive). + Where(entity.DatasourcePlatformCols.Status, "ACTIVE"). Count() if err != nil { return nil, err @@ -228,7 +226,7 @@ func (d *datasourcePlatformDao) BatchUpdateStatus(ctx context.Context, ids []int Data(map[string]interface{}{ entity.DatasourcePlatformCols.Status: status, entity.DatasourcePlatformCols.UpdatedBy: updatedBy, - entity.DatasourcePlatformCols.UpdatedAt: strconv.FormatInt(time.Now().Unix(), 10), + entity.DatasourcePlatformCols.UpdatedAt: time.Now(), }). WhereIn(entity.DatasourcePlatformCols.ID, ids). Update() diff --git a/dao/dict/api_interface_dao.go b/dao/dict/api_interface_dao.go index 007fd7d..64dde9e 100644 --- a/dao/dict/api_interface_dao.go +++ b/dao/dict/api_interface_dao.go @@ -84,7 +84,9 @@ func (d *apiInterfaceDao) buildListFilter(ctx context.Context, req *dto.ListApiI model.WhereLike(entity.ApiInterfaceCols.Name, "%"+req.Keyword+"%") model.WhereOrLike(entity.ApiInterfaceCols.Code, "%"+req.Keyword+"%") } - model.Where(entity.ApiInterfaceCols.PlatformId, req.PlatformId) + if req.PlatformId > 0 { + model.Where(entity.ApiInterfaceCols.PlatformId, req.PlatformId) + } model.Where(entity.ApiInterfaceCols.Name, req.Name) model.Where(entity.ApiInterfaceCols.Code, req.Code) model.Where(entity.ApiInterfaceCols.Method, req.Method) diff --git a/data-engine b/data-engine new file mode 100755 index 0000000..6207845 Binary files /dev/null and b/data-engine differ diff --git a/main.go b/main.go index 1154b34..47f3987 100644 --- a/main.go +++ b/main.go @@ -3,8 +3,13 @@ package main import ( "dataengine/controller/debug" "dataengine/controller/dict" + "dataengine/controller/public" + reportCtrl "dataengine/controller/report" syncCtrl "dataengine/controller/sync" syncSvc "dataengine/service/sync" + "os" + "os/signal" + "syscall" "gitea.redpowerfuture.com/red-future/common/http" "gitea.redpowerfuture.com/red-future/common/jaeger" @@ -12,11 +17,12 @@ import ( _ "github.com/gogf/gf/contrib/drivers/pgsql/v2" _ "github.com/gogf/gf/contrib/nosql/redis/v2" "github.com/gogf/gf/v2/frame/g" - "golang.org/x/net/context" + "github.com/gogf/gf/v2/os/gctx" + "github.com/sirupsen/logrus" ) func main() { - ctx := context.Background() + ctx := gctx.New() defer jaeger.ShutDown(ctx) // 启动自动同步（后台循环执行，首次全量后续增量） @@ -28,10 +34,20 @@ func main() { dict.DatasourcePlatform, // 平台同步引擎 syncCtrl.PlatformSyncController, + // 公共查询接口 + public.PublicQuery, + // 报表引擎 CRUD API + reportCtrl.ReportController, }) // 管理后台页面 g.Server().BindHandler("/admin", debug.DebugController.DebugPage) + // 报表引擎管理页面 + g.Server().BindHandler("/admin/report", reportCtrl.ReportAdminPage) - select {} + // 捕获退出信号，实现优雅关闭 + quit := make(chan os.Signal, 1) + signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM) + sig := <-quit + logrus.Infof("收到信号 %v，正在退出...", sig) } diff --git a/model/dto/dict/api_datasource_platform_dto.go b/model/dto/dict/api_datasource_platform_dto.go index 2fdcd78..3127b26 100644 --- a/model/dto/dict/api_datasource_platform_dto.go +++ b/model/dto/dict/api_datasource_platform_dto.go @@ -1,4 +1,4 @@ -package api_feature +package dict import ( "dataengine/consts/api-feature" diff --git a/model/dto/dict/api_interface_dto.go b/model/dto/dict/api_interface_dto.go index 26bfa16..ac88816 100644 --- a/model/dto/dict/api_interface_dto.go +++ b/model/dto/dict/api_interface_dto.go @@ -1,4 +1,4 @@ -package api_feature +package dict import ( "dataengine/consts/api-feature" diff --git a/model/dto/public/public_query_dto.go b/model/dto/public/public_query_dto.go new file mode 100644 index 0000000..9583b3a --- /dev/null +++ b/model/dto/public/public_query_dto.go @@ -0,0 +1,62 @@ +package public + +import ( + "github.com/gogf/gf/v2/frame/g" +) + +// QueryReq 公共查询请求 +type QueryReq struct { + g.Meta `path:"/public/query" method:"post" tags:"公共查询" summary:"公共数据查询" dc:"支持字段/表名/过滤/分组/分页的通用查询接口"` + Table string `json:"table" v:"required" dc:"表名（白名单限制）"` + Fields string `json:"fields" dc:"查询字段，逗号分隔，默认 *"` + Where map[string]interface{} `json:"where" dc:"过滤条件，支持操作符: =, !=, >, <, >=, <=, like, in, between"` + GroupBy string `json:"groupBy" dc:"分组字段，逗号分隔"` + OrderBy string `json:"orderBy" dc:"排序字段，如: create_time desc"` + Page int `json:"page" dc:"页码，默认1" d:"1"` + PageSize int `json:"pageSize" dc:"每页条数，默认20，最大100" d:"20"` +} + +// QueryRes 公共查询响应 +type QueryRes struct { + List []map[string]interface{} `json:"list" dc:"数据列表"` + Total int64 `json:"total" dc:"总数"` + Page int `json:"page" dc:"当前页码"` + Size int `json:"size" dc:"每页条数"` +} + +// TableListReq 获取可查询表列表请求 +type TableListReq struct { + g.Meta `path:"/public/tables" method:"get" tags:"公共查询" summary:"获取可查询表列表" dc:"获取配置了表结构的同步表列表"` +} + +// TableListRes 获取可查询表列表响应 +type TableListRes struct { + List []TableInfo `json:"list" dc:"表列表"` +} + +// TableInfo 表信息 +type TableInfo struct { + TableName string `json:"tableName" dc:"表名"` + PlatformName string `json:"platformName" dc:"平台名称"` + InterfaceName string `json:"interfaceName" dc:"接口名称"` + Columns []string `json:"columns" dc:"可用字段列表"` +} + +// ColumnListReq 获取表字段列表请求 +type ColumnListReq struct { + g.Meta `path:"/public/tables/{table}/columns" method:"get" tags:"公共查询" summary:"获取表字段列表" dc:"获取指定表的字段列表"` + Table string `json:"table" dc:"表名"` +} + +// ColumnListRes 获取表字段列表响应 +type ColumnListRes struct { + TableName string `json:"tableName" dc:"表名"` + Columns []Column `json:"columns" dc:"字段列表"` +} + +// Column 字段信息 +type Column struct { + Name string `json:"name" dc:"字段名"` + Type string `json:"type" dc:"字段类型"` + Comment string `json:"comment" dc:"字段说明"` +} diff --git a/model/entity/dict/api_datasource_platform.go b/model/entity/dict/api_datasource_platform.go index ecb92df..916ff28 100644 --- a/model/entity/dict/api_datasource_platform.go +++ b/model/entity/dict/api_datasource_platform.go @@ -36,10 +36,10 @@ type DatasourcePlatform struct { // 自定义认证配置 (JSONB) AuthConfig map[string]interface{} `orm:"auth_config" json:"authConfig" description:"自定义认证配置，支持各平台特有的认证方式"` - // 元数据 - CreatedBy string `orm:"created_by" json:"createdBy" description:"创建人"` + // 元数据（ORM tag 与数据库列名一致：creator/updater） + CreatedBy string `orm:"creator" json:"createdBy" description:"创建人"` CreatedAt *time.Time `orm:"created_at" json:"createdAt" description:"创建时间"` - UpdatedBy string `orm:"updated_by" json:"updatedBy" description:"更新人"` + UpdatedBy string `orm:"updater" json:"updatedBy" description:"更新人"` UpdatedAt *time.Time `orm:"updated_at" json:"updatedAt" description:"更新时间"` Version int `orm:"version" json:"version" description:"版本号(乐观锁)"` } @@ -64,9 +64,9 @@ type DatasourcePlatformCol struct { MaxRetries string RetryDelayMs string AuthConfig string - CreatedBy string + CreatedBy string // 对应 DB 列 creator CreatedAt string - UpdatedBy string + UpdatedBy string // 对应 DB 列 updater UpdatedAt string Version string } @@ -91,9 +91,9 @@ var DatasourcePlatformCols = DatasourcePlatformCol{ MaxRetries: "max_retries", RetryDelayMs: "retry_delay_ms", AuthConfig: "auth_config", - CreatedBy: "created_by", + CreatedBy: "creator", CreatedAt: "created_at", - UpdatedBy: "updated_by", + UpdatedBy: "updater", UpdatedAt: "updated_at", Version: "version", } diff --git a/scheduler/run_sync_task_log_task.go b/scheduler/run_sync_task_log_task.go index 37eb24a..2f98e16 100644 --- a/scheduler/run_sync_task_log_task.go +++ b/scheduler/run_sync_task_log_task.go @@ -1,132 +1,17 @@ package main import ( - "context" - "fmt" - "time" - - dao "dataengine/dao/copydata" - taskDto "dataengine/model/dto/copydata" syncSvc "dataengine/service/sync" - "gitea.redpowerfuture.com/red-future/common/beans" _ "github.com/gogf/gf/contrib/drivers/pgsql/v2" - "github.com/gogf/gf/v2/frame/g" "github.com/gogf/gf/v2/os/gctx" "github.com/sirupsen/logrus" ) -// CompensationScheduler 通用补偿调度器 -type CompensationScheduler struct { - interval time.Duration -} - -// NewCompensationScheduler 创建调度器 -func NewCompensationScheduler() *CompensationScheduler { - ctx := gctx.New() - sec := g.Cfg().MustGet(ctx, "sync.compensation_interval_seconds", 300).Int() - if sec < 10 { - sec = 300 - } - return &CompensationScheduler{interval: time.Duration(sec) * time.Second} -} - -// RunOnce 执行一次补偿 -func (s *CompensationScheduler) RunOnce() { - ctx := gctx.New() - ctx = context.WithValue(ctx, "user", &beans.User{UserName: "admin", TenantId: 1}) - - logrus.Info("=== 开始补偿扫描 ===") - - // 查询所有 failed 状态的任务，不限类型 - tasks, err := dao.SyncTaskLog.QueryFailedTasks(ctx, &taskDto.QueryFailedTasksReq{ - Status: []string{"failed"}, - Limit: 50, - }) - if err != nil { - logrus.Errorf("查询失败任务异常: %v", err) - return - } - if len(tasks) == 0 { - logrus.Info("当前没有需要补偿的任务") - return - } - - logrus.Infof("发现 %d 个失败任务", len(tasks)) - - for _, task := range tasks { - if task.RetryCount >= task.MaxRetry { - logrus.Warnf("任务 %s 已达最大重试次数 %d", task.TaskID, task.MaxRetry) - dao.SyncTaskLog.Update(ctx, &taskDto.UpdateSyncTaskLogReq{ - ID: task.Id, - Status: "manual_review", - ErrorMessage: fmt.Sprintf("已达最大重试次数 %d", task.MaxRetry), - }) - continue - } - - platformCode := task.PlatformCode - interfaceCode := task.InterfaceCode - if platformCode == "" || interfaceCode == "" { - logrus.Warnf("任务 %s 缺少 platform_code 或 interface_code，跳过", task.TaskID) - continue - } - - logrus.Infof("补偿: %s/%s (第 %d 次)", platformCode, interfaceCode, task.RetryCount+1) - - // 更新状态为 retrying - retryCount := task.RetryCount + 1 - dao.SyncTaskLog.Update(ctx, &taskDto.UpdateSyncTaskLogReq{ - ID: task.Id, - Status: "retrying", - RetryCount: &retryCount, - }) - - // 执行补偿（增量同步） - _, err := syncSvc.SyncByConfig(ctx, platformCode, interfaceCode, false) - if err != nil { - logrus.Errorf("补偿失败: %v", err) - // 更新状态为 failed，设置下次重试时间 - backoff := []int{5, 15, 30, 60, 120} - waitMin := 5 - if retryCount <= len(backoff) { - waitMin = backoff[retryCount-1] - } else { - waitMin = backoff[len(backoff)-1] - } - nextRetry := time.Now().Add(time.Duration(waitMin) * time.Minute) - dao.SyncTaskLog.Update(ctx, &taskDto.UpdateSyncTaskLogReq{ - ID: task.Id, - Status: "failed", - ErrorMessage: err.Error(), - ErrorCode: "COMPENSATION_FAILED", - NextRetryTime: nextRetry, - }) - } else { - logrus.Infof("补偿成功: %s/%s", platformCode, interfaceCode) - now := time.Now() - dao.SyncTaskLog.Update(ctx, &taskDto.UpdateSyncTaskLogReq{ - ID: task.Id, - Status: "success", - CompletedAt: now, - }) - } - } - - logrus.Info("=== 补偿扫描完成 ===") -} - -// Start 启动 -func (s *CompensationScheduler) Start() { - logrus.Infof("补偿调度器启动，间隔: %v", s.interval) - s.RunOnce() - ticker := time.NewTicker(s.interval) - defer ticker.Stop() - for range ticker.C { - s.RunOnce() - } -} - func main() { - NewCompensationScheduler().Start() + ctx := gctx.New() + logrus.Info("独立补偿调度器启动") + syncSvc.StartCompensation(ctx) + // StartCompensation 内部有永续循环，不会返回 + select {} } diff --git a/service/dict/api_datasource_platform_service.go b/service/dict/api_datasource_platform_service.go index f00a6db..5549a49 100644 --- a/service/dict/api_datasource_platform_service.go +++ b/service/dict/api_datasource_platform_service.go @@ -79,9 +79,9 @@ func (s *datasourcePlatformService) List(ctx context.Context, req *dto.ListDatas MaxRetries: item.MaxRetries, RetryDelayMs: item.RetryDelayMs, CreatedBy: item.CreatedBy, - CreatedAt: item.CreatedAt.Unix(), + CreatedAt: s.safeUnix(item.CreatedAt), UpdatedBy: item.UpdatedBy, - UpdatedAt: item.UpdatedAt.Unix(), + UpdatedAt: s.safeUnix(item.UpdatedAt), }) } @@ -294,6 +294,14 @@ func (s *datasourcePlatformService) getAuthTypeName(authType string) string { return authType } +// safeUnix 安全地从 *time.Time 获取 Unix 时间戳，nil 返回 0 +func (s *datasourcePlatformService) safeUnix(t *time.Time) int64 { + if t == nil { + return 0 + } + return t.Unix() +} + // BatchUpdateStatus 批量更新平台状态 func (s *datasourcePlatformService) BatchUpdateStatus(ctx context.Context, ids []int64, status string, updatedBy string) (err error) { if len(ids) == 0 { diff --git a/service/dict/api_interface_service.go b/service/dict/api_interface_service.go index 7d0022a..9b2fe50 100644 --- a/service/dict/api_interface_service.go +++ b/service/dict/api_interface_service.go @@ -7,6 +7,8 @@ import ( dto "dataengine/model/dto/dict" entity "dataengine/model/entity/dict" "errors" + + "github.com/gogf/gf/v2/os/gtime" ) type apiInterfaceService struct{} @@ -85,8 +87,8 @@ func (s *apiInterfaceService) List(ctx context.Context, req *dto.ListApiInterfac Method: item.Method, Status: item.Status, StatusName: s.getStatusName(item.Status), - CreatedAt: item.CreatedAt.Unix(), - UpdatedAt: item.UpdatedAt.Unix(), + CreatedAt: safeUnix(item.CreatedAt), + UpdatedAt: safeUnix(item.UpdatedAt), }) } @@ -181,3 +183,11 @@ func (s *apiInterfaceService) getStatusName(status consts.PlatformStatus) string } return string(status) } + +// safeUnix 安全地从 *gtime.Time 获取 Unix 时间戳，nil 返回 0 +func safeUnix(t *gtime.Time) int64 { + if t == nil { + return 0 + } + return t.Unix() +} diff --git a/service/public/public_query_service.go b/service/public/public_query_service.go new file mode 100644 index 0000000..33f9e16 --- /dev/null +++ b/service/public/public_query_service.go @@ -0,0 +1,524 @@ +package public + +import ( + "context" + "fmt" + "regexp" + "strings" + + "dataengine/model/dto/public" + "dataengine/model/entity/dict" + + "gitea.redpowerfuture.com/red-future/common/db/gfdb" + "github.com/gogf/gf/v2/util/gconv" +) + +var PublicQuery = new(publicQueryService) + +// tableColumnsCache 表定义缓存 +var tableColumnsCache = make(map[string][]string) + +type publicQueryService struct{} + +// Query 执行公共查询 +func (s *publicQueryService) Query(ctx context.Context, req *public.QueryReq) (res *public.QueryRes, err error) { + // 1. 验证表名白名单 + if err = s.validateTable(ctx, req.Table); err != nil { + return nil, err + } + + // 2. 验证字段白名单 + allowedFields, err := s.getAllowedFields(ctx, req.Table) + if err != nil { + return nil, err + } + + // 3. 构建 SELECT 部分 + selectFields := "*" + if req.Fields != "" { + selectFields, err = s.buildSelectFields(req.Fields, allowedFields) + if err != nil { + return nil, err + } + } + + // 4. 构建 WHERE 条件 + whereClause, whereArgs, err := s.buildWhereClause(req.Where, allowedFields) + if err != nil { + return nil, err + } + + // 5. 构建 GROUP BY + groupByClause, err := s.buildGroupBy(req.GroupBy, allowedFields) + if err != nil { + return nil, err + } + + // 6. 强制租户过滤 + tenantClause := s.buildTenantClause(allowedFields) + + // 7. 组合完整 WHERE + fullWhere := tenantClause + if whereClause != "" { + if fullWhere != "" { + fullWhere += " AND " + whereClause + } else { + fullWhere = whereClause + } + } + + // 8. 校验分页参数 + if req.Page < 1 { + req.Page = 1 + } + if req.PageSize < 1 { + req.PageSize = 20 + } + if req.PageSize > 100 { + req.PageSize = 100 + } + offset := (req.Page - 1) * req.PageSize + + // 9. 构建 ORDER BY + orderByClause, err := s.buildOrderBy(req.OrderBy, allowedFields) + if err != nil { + return nil, err + } + + // 10. 统计总数 + countSql := fmt.Sprintf("SELECT COUNT(*) FROM %s", req.Table) + if fullWhere != "" { + countSql += " WHERE " + fullWhere + } + if groupByClause != "" { + countSql = fmt.Sprintf("SELECT COUNT(*) FROM (SELECT 1 FROM %s WHERE %s GROUP BY %s) AS t", + req.Table, fullWhere, groupByClause) + } + + result, err := gfdb.DB(ctx).GetAll(ctx, countSql, whereArgs...) + if err != nil { + return nil, fmt.Errorf("统计总数失败: %v", err) + } + var total int64 + if result.Len() > 0 { + total = result[0]["count"].Int64() + } + + // 11. 查询数据 + querySql := fmt.Sprintf("SELECT %s FROM %s", selectFields, req.Table) + if fullWhere != "" { + querySql += " WHERE " + fullWhere + } + if groupByClause != "" { + querySql += " GROUP BY " + groupByClause + } + if orderByClause != "" { + querySql += " ORDER BY " + orderByClause + } + querySql += fmt.Sprintf(" LIMIT %d OFFSET %d", req.PageSize, offset) + + dataResult, err := gfdb.DB(ctx).GetAll(ctx, querySql, whereArgs...) + if err != nil { + return nil, fmt.Errorf("查询数据失败: %v", err) + } + + var list []map[string]interface{} + if dataResult.Len() > 0 { + list = dataResult.List() + } + + return &public.QueryRes{ + List: list, + Total: total, + Page: req.Page, + Size: req.PageSize, + }, nil +} + +// GetTableList 获取可查询表列表 +func (s *publicQueryService) GetTableList(ctx context.Context) (*public.TableListRes, error) { + var ifaces []dict.ApiInterface + err := gfdb.DB(ctx).Model(ctx, "api_interface"). + Where("table_definition IS NOT NULL"). + Where("table_definition->>'table_name' != ''"). + Where("status", "active"). + Scan(&ifaces) + if err != nil { + return nil, fmt.Errorf("查询表列表失败: %v", err) + } + + // 查询平台名称 + var platforms []dict.DatasourcePlatform + _ = gfdb.DB(ctx).Model(ctx, "api_datasource_platform").Scan(&platforms) + platformMap := make(map[int64]string) + for _, p := range platforms { + platformMap[p.ID] = p.PlatformName + } + + var list []public.TableInfo + for _, iface := range ifaces { + tableName := s.getStringFromMap(iface.TableDefinition, "table_name") + if tableName == "" { + continue + } + + columns := s.extractColumnsFromMap(iface.TableDefinition) + list = append(list, public.TableInfo{ + TableName: tableName, + PlatformName: platformMap[iface.PlatformId], + InterfaceName: iface.Name, + Columns: columns, + }) + } + + return &public.TableListRes{List: list}, nil +} + +// GetColumnList 获取表字段列表 +func (s *publicQueryService) GetColumnList(ctx context.Context, tableName string) (*public.ColumnListRes, error) { + if err := s.validateTable(ctx, tableName); err != nil { + return nil, err + } + + columns, err := s.getColumnDetails(ctx, tableName) + if err != nil { + return nil, err + } + + return &public.ColumnListRes{ + TableName: tableName, + Columns: columns, + }, nil +} + +// validateTable 验证表名白名单 +func (s *publicQueryService) validateTable(ctx context.Context, tableName string) error { + if tableName == "" { + return fmt.Errorf("表名不能为空") + } + + // 表名格式校验 + if matched, _ := regexp.MatchString(`^[a-zA-Z_][a-zA-Z0-9_]*$`, tableName); !matched { + return fmt.Errorf("表名格式非法，只允许字母、数字、下划线") + } + + // 禁止系统表 + systemTables := []string{"pg_catalog", "information_schema"} + for _, t := range systemTables { + if strings.HasPrefix(strings.ToLower(tableName), t) { + return fmt.Errorf("禁止查询系统表") + } + } + + // 检查白名单 + count, err := gfdb.DB(ctx).Model(ctx, "api_interface"). + Where("table_definition->>'table_name' = ?", tableName). + Where("status", "active"). + Count() + if err != nil { + return fmt.Errorf("表名验证失败: %v", err) + } + if count == 0 { + return fmt.Errorf("表 [%s] 不在可查询白名单中", tableName) + } + + return nil +} + +// getAllowedFields 获取表允许的字段 +func (s *publicQueryService) getAllowedFields(ctx context.Context, tableName string) ([]string, error) { + if cols, ok := tableColumnsCache[tableName]; ok { + return cols, nil + } + + var iface dict.ApiInterface + _, err := gfdb.DB(ctx).Model(ctx, "api_interface"). + Where("table_definition->>'table_name' = ?", tableName). + Where("status", "active"). + One(&iface) + if err != nil { + return nil, fmt.Errorf("获取表字段失败: %v", err) + } + + columns := s.extractColumnsFromMap(iface.TableDefinition) + tableColumnsCache[tableName] = columns + return columns, nil +} + +// extractColumnsFromMap 从 map 中提取字段 +func (s *publicQueryService) extractColumnsFromMap(tableDef map[string]interface{}) []string { + var columns []string + if cols, ok := tableDef["columns"].([]interface{}); ok { + for _, c := range cols { + if col, ok := c.(map[string]interface{}); ok { + if name, ok := col["name"].(string); ok { + columns = append(columns, name) + } + } + } + } + return columns +} + +// buildSelectFields 构建 SELECT 字段 +func (s *publicQueryService) buildSelectFields(fields string, allowedFields []string) (string, error) { + allowedMap := make(map[string]bool) + for _, f := range allowedFields { + allowedMap[strings.ToLower(f)] = true + } + allowedMap["id"] = true + allowedMap["tenant_id"] = true + allowedMap["created_at"] = true + allowedMap["updated_at"] = true + allowedMap["raw_data"] = true + + var result []string + for _, f := range strings.Split(fields, ",") { + f = strings.TrimSpace(f) + if f == "" { + continue + } + if strings.Contains(f, " ") || strings.Contains(f, "(") { + result = append(result, f) + continue + } + fLower := strings.ToLower(f) + if !allowedMap[fLower] { + return "", fmt.Errorf("字段 [%s] 不在允许列表中", f) + } + result = append(result, f) + } + + if len(result) == 0 { + return "*", nil + } + return strings.Join(result, ", "), nil +} + +// buildWhereClause 构建 WHERE 条件 +func (s *publicQueryService) buildWhereClause(where map[string]interface{}, allowedFields []string) (string, []interface{}, error) { + if len(where) == 0 { + return "", nil, nil + } + + allowedMap := make(map[string]bool) + for _, f := range allowedFields { + allowedMap[strings.ToLower(f)] = true + } + allowedMap["tenant_id"] = true + + var conditions []string + var args []interface{} + + for field, value := range where { + fieldLower := strings.ToLower(field) + if !allowedMap[fieldLower] { + return "", nil, fmt.Errorf("字段 [%s] 不在允许列表中", field) + } + + // 处理操作符后缀 + opSuffixes := []struct { + suffix string + format string + like bool + }{ + {"_eq", "%s = ?", false}, + {"_ne", "%s != ?", false}, + {"_gt", "%s > ?", false}, + {"_lt", "%s < ?", false}, + {"_ge", "%s >= ?", false}, + {"_le", "%s <= ?", false}, + {"_like", "%s LIKE ?", true}, + } + + matched := false + for _, op := range opSuffixes { + if strings.HasSuffix(fieldLower, op.suffix) { + cleanField := field[:len(field)-len(op.suffix)] + conditions = append(conditions, fmt.Sprintf(op.format, cleanField)) + if op.like { + args = append(args, "%"+gconv.String(value)+"%") + } else { + args = append(args, value) + } + matched = true + break + } + } + if matched { + continue + } + + // 处理 _in + if strings.HasSuffix(fieldLower, "_in") { + cleanField := field[:len(field)-3] + if arr, ok := value.([]interface{}); ok { + placeholders := make([]string, len(arr)) + for i, v := range arr { + placeholders[i] = "?" + args = append(args, v) + } + conditions = append(conditions, fmt.Sprintf("%s IN (%s)", cleanField, strings.Join(placeholders, ","))) + } else if str, ok := value.(string); ok { + parts := strings.Split(str, ",") + placeholders := make([]string, len(parts)) + for i, p := range parts { + placeholders[i] = "?" + args = append(args, strings.TrimSpace(p)) + } + conditions = append(conditions, fmt.Sprintf("%s IN (%s)", cleanField, strings.Join(placeholders, ","))) + } + continue + } + + // 处理 _between + if strings.HasSuffix(fieldLower, "_between") { + cleanField := field[:len(field)-8] + if arr, ok := value.([]interface{}); ok && len(arr) >= 2 { + conditions = append(conditions, fmt.Sprintf("%s BETWEEN ? AND ?", cleanField)) + args = append(args, arr[0], arr[1]) + } else if arr, ok := value.([]string); ok && len(arr) >= 2 { + conditions = append(conditions, fmt.Sprintf("%s BETWEEN ? AND ?", cleanField)) + args = append(args, arr[0], arr[1]) + } + continue + } + + // 默认等于 + conditions = append(conditions, fmt.Sprintf("%s = ?", field)) + args = append(args, value) + } + + return strings.Join(conditions, " AND "), args, nil +} + +// buildGroupBy 构建 GROUP BY +func (s *publicQueryService) buildGroupBy(groupBy string, allowedFields []string) (string, error) { + if groupBy == "" { + return "", nil + } + + allowedMap := make(map[string]bool) + for _, f := range allowedFields { + allowedMap[strings.ToLower(f)] = true + } + + var fields []string + for _, f := range strings.Split(groupBy, ",") { + f = strings.TrimSpace(f) + if f == "" { + continue + } + if !allowedMap[strings.ToLower(f)] { + return "", fmt.Errorf("分组字段 [%s] 不在允许列表中", f) + } + fields = append(fields, f) + } + + return strings.Join(fields, ", "), nil +} + +// buildOrderBy 构建 ORDER BY +func (s *publicQueryService) buildOrderBy(orderBy string, allowedFields []string) (string, error) { + if orderBy == "" { + return "", nil + } + + allowedMap := make(map[string]bool) + for _, f := range allowedFields { + allowedMap[strings.ToLower(f)] = true + } + allowedMap["id"] = true + allowedMap["created_at"] = true + allowedMap["updated_at"] = true + + var clauses []string + for _, part := range strings.Split(orderBy, ",") { + part = strings.TrimSpace(part) + if part == "" { + continue + } + + parts := strings.Fields(part) + if len(parts) == 0 { + continue + } + + field := parts[0] + dir := "ASC" + if len(parts) > 1 { + if strings.ToUpper(parts[1]) == "DESC" { + dir = "DESC" + } + } + + if !allowedMap[strings.ToLower(field)] { + return "", fmt.Errorf("排序字段 [%s] 不在允许列表中", field) + } + field = regexp.MustCompile(`[^a-zA-Z0-9_]`).ReplaceAllString(field, "") + clauses = append(clauses, field+" "+dir) + } + + return strings.Join(clauses, ", "), nil +} + +// buildTenantClause 构建租户过滤条件 +func (s *publicQueryService) buildTenantClause(allowedFields []string) string { + for _, f := range allowedFields { + if strings.ToLower(f) == "tenant_id" { + return "tenant_id = 1" + } + } + return "" +} + +// getColumnDetails 获取表字段详情 +func (s *publicQueryService) getColumnDetails(ctx context.Context, tableName string) ([]public.Column, error) { + var iface dict.ApiInterface + _, err := gfdb.DB(ctx).Model(ctx, "api_interface"). + Where("table_definition->>'table_name' = ?", tableName). + Where("status", "active"). + One(&iface) + if err != nil { + return nil, fmt.Errorf("获取表字段详情失败: %v", err) + } + + var columns []public.Column + if cols, ok := iface.TableDefinition["columns"].([]interface{}); ok { + for _, c := range cols { + if col, ok := c.(map[string]interface{}); ok { + columns = append(columns, public.Column{ + Name: gconv.String(col["name"]), + Type: gconv.String(col["type"]), + Comment: gconv.String(col["comment"]), + }) + } + } + } + + columns = append(columns, public.Column{Name: "id", Type: "BIGINT", Comment: "主键ID"}) + columns = append(columns, public.Column{Name: "tenant_id", Type: "BIGINT", Comment: "租户ID"}) + columns = append(columns, public.Column{Name: "created_at", Type: "TIMESTAMP", Comment: "创建时间"}) + columns = append(columns, public.Column{Name: "updated_at", Type: "TIMESTAMP", Comment: "更新时间"}) + columns = append(columns, public.Column{Name: "raw_data", Type: "JSONB", Comment: "原始数据"}) + + return columns, nil +} + +// getStringFromMap 从 map 中获取字符串值 +func (s *publicQueryService) getStringFromMap(data map[string]interface{}, key string) string { + if v, ok := data[key].(string); ok { + return v + } + return "" +} + +// ClearTableCache 清除表缓存 +func (s *publicQueryService) ClearTableCache() { + tableColumnsCache = make(map[string][]string) +} + +// InvalidateTableCache 失效指定表的缓存 +func (s *publicQueryService) InvalidateTableCache(tableName string) { + delete(tableColumnsCache, tableName) +} diff --git a/service/sync/api_client.go b/service/sync/api_client.go index 864eed4..90e2085 100644 --- a/service/sync/api_client.go +++ b/service/sync/api_client.go @@ -29,7 +29,7 @@ type ApiResult struct { type ApiClient struct { config *PlatformConfig client *http.Client - rateLimiter <-chan time.Time // 限流 ticker + rateLimiter *time.Ticker // 限流 ticker，可被 GC } // NewApiClient 创建客户端 @@ -38,14 +38,22 @@ func NewApiClient(config *PlatformConfig) *ApiClient { if config.RequestTimeoutMs > 0 { timeout = time.Duration(config.RequestTimeoutMs) * time.Millisecond } + transport := &http.Transport{ + MaxIdleConns: 100, + MaxIdleConnsPerHost: 20, + IdleConnTimeout: 90 * time.Second, + } ac := &ApiClient{ config: config, - client: &http.Client{Timeout: timeout}, + client: &http.Client{ + Timeout: timeout, + Transport: transport, + }, } // 初始化限流 if config.RateLimitPerMinute > 0 { interval := time.Minute / time.Duration(config.RateLimitPerMinute) - ac.rateLimiter = time.Tick(interval) + ac.rateLimiter = time.NewTicker(interval) logrus.Infof("限流已启用: %d 次/分钟, 间隔 %v", config.RateLimitPerMinute, interval) } return ac @@ -61,6 +69,13 @@ func (c *ApiClient) PostJSON(ctx context.Context, path string, body interface{}) return c.doRequest(ctx, "POST", path, body, false) } +// Close 释放客户端资源（限流 ticker） +func (c *ApiClient) Close() { + if c.rateLimiter != nil { + c.rateLimiter.Stop() + } +} + // Request 通用请求方法（支持 GET/POST，支持参数在 query 或 body） func (c *ApiClient) Request(ctx context.Context, method, path string, params map[string]interface{}, paramsInQuery bool) (*ApiResult, error) { if paramsInQuery { @@ -99,8 +114,9 @@ func (c *ApiClient) execute(ctx context.Context, method, path string, body inter // 限流等待 if c.rateLimiter != nil { select { - case <-c.rateLimiter: + case <-c.rateLimiter.C: case <-ctx.Done(): + c.rateLimiter.Stop() return nil, ctx.Err() } } @@ -112,8 +128,13 @@ func (c *ApiClient) execute(ctx context.Context, method, path string, body inter fullURL = c.applyAuthURL(fullURL) var reqBody io.Reader + var reqBodyBytes []byte if body != nil && !paramsInQuery { - b, _ := json.Marshal(body) + b, err := json.Marshal(body) + if err != nil { + return nil, fmt.Errorf("JSON序列化请求体失败: %w", err) + } + reqBodyBytes = b reqBody = bytes.NewBuffer(b) } @@ -133,7 +154,7 @@ func (c *ApiClient) execute(ctx context.Context, method, path string, body inter return nil, fmt.Errorf("创建请求失败: %w", err) } - c.applyAuthHeader(req) + c.applyAuthHeader(req, reqBodyBytes) req.Header.Set("User-Agent", "data-engine/1.0") if body != nil && !paramsInQuery { req.Header.Set("Content-Type", "application/json") @@ -145,7 +166,10 @@ func (c *ApiClient) execute(ctx context.Context, method, path string, body inter } defer resp.Body.Close() - respBody, _ := io.ReadAll(resp.Body) + respBody, err := io.ReadAll(resp.Body) + if err != nil { + return nil, fmt.Errorf("读取响应体失败: %w", err) + } result := &ApiResult{Body: respBody, DurationMs: time.Since(start).Milliseconds()} if resp.StatusCode >= 400 { @@ -157,7 +181,11 @@ func (c *ApiClient) execute(ctx context.Context, method, path string, body inter // buildQueryURL 将 params 拼接到 URL 查询参数中 // 支持数组/对象类型的值自动 JSON 序列化 + URL 编码 func (c *ApiClient) buildQueryURL(rawURL string, params map[string]interface{}) string { - parsed, _ := url.Parse(rawURL) + parsed, err := url.Parse(rawURL) + if err != nil || parsed == nil { + logrus.Errorf("buildQueryURL: 解析 URL 失败: %v", err) + return rawURL + } q := parsed.Query() for k, v := range params { @@ -224,7 +252,11 @@ func (c *ApiClient) applyAuthURL(rawURL string) string { return rawURL } - parsed, _ := url.Parse(rawURL) + parsed, err := url.Parse(rawURL) + if err != nil || parsed == nil { + logrus.Errorf("applyAuthURL: 解析 URL 失败: %v", err) + return rawURL + } q := parsed.Query() if tokenInQuery && token != "" { q.Set(queryKey, token) @@ -236,10 +268,16 @@ func (c *ApiClient) applyAuthURL(rawURL string) string { return parsed.String() } -func (c *ApiClient) applyAuthHeader(req *http.Request) { +func (c *ApiClient) applyAuthHeader(req *http.Request, bodyBytes []byte) { cfg := c.config.AuthConfig token := c.config.AccessToken + // APP_SIGNATURE 认证：app-id + signature 头部（如钉钉智能薪酬） + if c.config.AuthType == "APP_SIGNATURE" { + c.applyAppSignatureAuth(req, bodyBytes) + return + } + if cfg != nil { if tiq, _ := cfg["token_in_query"].(bool); tiq { return @@ -251,9 +289,9 @@ func (c *ApiClient) applyAuthHeader(req *http.Request) { if cfg != nil { if h, ok := cfg["header_name"].(string); ok { - f := cfg["header_format"].(string) - if f == "" { - f = "{token}" + f := "{token}" + if fv, ok2 := cfg["header_format"].(string); ok2 { + f = fv } req.Header.Set(h, strings.ReplaceAll(f, "{token}", token)) return @@ -268,6 +306,73 @@ func (c *ApiClient) applyAuthHeader(req *http.Request) { } } +// applyAppSignatureAuth 设置 app-id + signature 认证头部 +func (c *ApiClient) applyAppSignatureAuth(req *http.Request, bodyBytes []byte) { + cfg := c.config.AuthConfig + if cfg == nil { + return + } + + // 1. 设置 app-id 头部 + appIdHeader := "app-id" + if h, _ := cfg["app_id_header"].(string); h != "" { + appIdHeader = h + } + appId := c.config.AppKey + if appId == "" { + if aid, _ := cfg["app_id"].(string); aid != "" { + appId = aid + } + } + if appId != "" { + req.Header.Set(appIdHeader, appId) + } + + // 2. 计算签名并设置 signature 头部 + signHeader := "signature" + if h, _ := cfg["sign_header"].(string); h != "" { + signHeader = h + } + + secret := c.config.AppSecret + + signAlgo := "md5_upper_body" + if a, _ := cfg["sign_algorithm"].(string); a != "" { + signAlgo = a + } + + sig := computeBodySignature(bodyBytes, secret, signAlgo) + if sig != "" { + req.Header.Set(signHeader, sig) + } +} + +// computeBodySignature 计算基于请求体的签名 +// 支持的算法: +// - md5_upper_body: MD5(body_string + secret) 大写（默认，钉钉智能薪酬） +// - md5_body: MD5(body_string + secret) 小写 +func computeBodySignature(bodyBytes []byte, secret, algo string) string { + if secret == "" { + return "" + } + bodyStr := "" + if len(bodyBytes) > 0 { + bodyStr = string(bodyBytes) + } + switch algo { + case "md5_body", "md5_upper_body": + h := md5.Sum([]byte(bodyStr + secret)) + sig := hex.EncodeToString(h[:]) + if algo == "md5_upper_body" { + sig = strings.ToUpper(sig) + } + return sig + default: + logrus.Warnf("未知签名算法: %s", algo) + return "" + } +} + func generateNonce() string { nanoPart := time.Now().UnixNano() % 1000000000000 r, _ := rand.Int(rand.Reader, big.NewInt(10000)) @@ -293,7 +398,11 @@ func (c *ApiClient) applySignature(rawURL string, body interface{}, paramsInQuer return rawURL } - parsed, _ := url.Parse(rawURL) + parsed, err := url.Parse(rawURL) + if err != nil || parsed == nil { + logrus.Errorf("applySignature: 解析 URL 失败: %v", err) + return rawURL + } q := parsed.Query() // 收集所有参数并按 key 排序 diff --git a/service/sync/compensation.go b/service/sync/compensation.go index 060d273..7580c4f 100644 --- a/service/sync/compensation.go +++ b/service/sync/compensation.go @@ -8,7 +8,7 @@ import ( dao "dataengine/dao/copydata" taskDto "dataengine/model/dto/copydata" - "gitea.com/red-future/common/beans" + "gitea.redpowerfuture.com/red-future/common/beans" "github.com/gogf/gf/v2/frame/g" "github.com/sirupsen/logrus" ) diff --git a/service/sync/data_writer.go b/service/sync/data_writer.go index e07ee7a..6644fa9 100644 --- a/service/sync/data_writer.go +++ b/service/sync/data_writer.go @@ -4,7 +4,7 @@ import ( "context" "time" - "gitea.com/red-future/common/db/gfdb" + "gitea.redpowerfuture.com/red-future/common/db/gfdb" "github.com/sirupsen/logrus" ) @@ -19,12 +19,8 @@ func InsertRows(ctx context.Context, tableName string, conflictKeys []string, ro if rows[i] == nil { rows[i] = make(map[string]interface{}) } - if _, ok := rows[i]["created_at"]; !ok { - rows[i]["created_at"] = now - } - if _, ok := rows[i]["updated_at"]; !ok { - rows[i]["updated_at"] = now - } + // 始终覆盖 updated_at；不设置 created_at 让数据库维护首次值（upsert 时不会覆盖） + rows[i]["updated_at"] = now } batchSize := 100 diff --git a/service/sync/dynamic_sync.go b/service/sync/dynamic_sync.go index aee26f3..1c1c53e 100644 --- a/service/sync/dynamic_sync.go +++ b/service/sync/dynamic_sync.go @@ -4,6 +4,7 @@ import ( "context" "encoding/json" "fmt" + "strconv" "strings" "sync" "time" @@ -13,7 +14,7 @@ import ( taskDto "dataengine/model/dto/copydata" entity "dataengine/model/entity/dict" - "gitea.com/red-future/common/db/gfdb" + "gitea.redpowerfuture.com/red-future/common/db/gfdb" "github.com/sirupsen/logrus" ) @@ -38,8 +39,20 @@ type PrefetchConfig struct { ValueField string `json:"value_field"` } +// RecursiveConfig 递归遍历配置（如钉钉部门树） +type RecursiveConfig struct { + KeyField string `json:"key_field"` + TargetParam string `json:"target_param"` +} + // SyncByConfig 执行同步 func SyncByConfig(ctx context.Context, platformCode, interfaceCode string, isFullSync bool) (*SyncResult, error) { + // 创建超时 context 防止单次同步卡死 + timeoutMin := GetSyncTimeout(ctx) + timeoutCtx, cancel := context.WithTimeout(ctx, time.Duration(timeoutMin)*time.Minute) + defer cancel() + ctx = timeoutCtx + // 内存锁：防止同一个接口被并发执行（两个调度周期重叠时跳过） lockKey := platformCode + "/" + interfaceCode if _, loaded := syncRunningMap.LoadOrStore(lockKey, true); loaded { @@ -93,11 +106,16 @@ func SyncByConfig(ctx context.Context, platformCode, interfaceCode string, isFul markSyncRunning(ctx, platformCode, interfaceCode, lastSyncTime) api := NewApiClient(platform) + defer api.Close() prefetch := parsePrefetchConfig(iface.RequestConfig) if prefetch != nil { return syncWithPrefetch(ctx, api, platform, iface, ifaces, td, prefetch, isFullSync, lastSyncTime, start) } + recursive := parseRecursiveConfig(iface.RequestConfig) + if recursive != nil { + return syncRecursive(ctx, api, platform, iface, td, recursive, start) + } return syncSingleAPI(ctx, api, platform, iface, td, isFullSync, lastSyncTime, start) } @@ -119,6 +137,8 @@ func syncSingleAPI(ctx context.Context, api *ApiClient, platform *PlatformConfig pageSize := GetSyncPageSize(ctx) if ps, ok := iface.RequestConfig["page_size"].(float64); ok { pageSize = int(ps) + } else if ps, ok := iface.RequestConfig["pageSize"].(float64); ok { + pageSize = int(ps) } taskType := "incremental" @@ -129,14 +149,19 @@ func syncSingleAPI(ctx context.Context, api *ApiClient, platform *PlatformConfig inQuery := paramsInQuery(iface) method := string(iface.Method) - // 游标分页首次请求需要 cursor=""（通过 extraParams 覆盖 buildReqBody 的 page=1 赋值） + // 游标分页首次请求需要处理初始游标值 firstExtra := map[string]interface{}{} if isCursorPagination(iface) { cp := "cursor" if p, ok := iface.RequestConfig["page_param"].(string); ok && p != "" { cp = p } - firstExtra[cp] = "" + // 支持 initial_cursor 配置（如钉钉HRM首次传 0） + if icv, ok := iface.RequestConfig["initial_cursor"]; ok { + firstExtra[cp] = icv + } else { + firstExtra[cp] = "" + } } body := buildReqBody(iface, 1, pageSize, lastSyncTime, firstExtra) resp, err := api.Request(ctx, method, iface.Url, body, inQuery) @@ -151,6 +176,8 @@ func syncSingleAPI(ctx context.Context, api *ApiClient, platform *PlatformConfig return nil, err } + injectRowFields(rows, body, iface.RequestConfig) + result := &SyncResult{TableName: td.TableName, TotalPages: totalPages} inserted, _ := savePage(ctx, td, rows) result.InsertedRows += inserted @@ -185,6 +212,7 @@ func syncSingleAPI(ctx context.Context, api *ApiClient, platform *PlatformConfig } nextCursor = nc + injectRowFields(rows, body, iface.RequestConfig) inserted, _ = savePage(ctx, td, rows) result.InsertedRows += inserted result.TotalRows += len(rows) @@ -194,22 +222,72 @@ func syncSingleAPI(ctx context.Context, api *ApiClient, platform *PlatformConfig result.TotalPages++ time.Sleep(100 * time.Millisecond) } + } else if iface.ResponseConfig != nil { + // hasMore 分页（如钉钉 offset/size + hasMore） + if hf, _ := iface.ResponseConfig["has_more_field"].(string); hf != "" { + for page := 2; hasMoreCheck(resp.Body, hf); page++ { + body := buildReqBody(iface, page, pageSize, lastSyncTime, nil) + resp2, e2 := api.Request(ctx, method, iface.Url, body, inQuery) + if e2 != nil { + logrus.Errorf("第 %d 页请求失败: %v", page, e2) + break + } + rows2, _, mt2, _, pe2 := parseRespExt(resp2.Body, iface.ResponseConfig) + if pe2 != nil { + logrus.Errorf("第 %d 页解析失败: %v", page, pe2) + break + } + injectRowFields(rows2, body, iface.RequestConfig) + inserted2, _ := savePage(ctx, td, rows2) + result.InsertedRows += inserted2 + result.TotalRows += len(rows2) + if mt2 > maxTime { + maxTime = mt2 + } + resp = resp2 + time.Sleep(100 * time.Millisecond) + } + } else { + // 普通分页 + for page := 2; page <= totalPages; page++ { + body := buildReqBody(iface, page, pageSize, lastSyncTime, nil) + resp, err = api.Request(ctx, method, iface.Url, body, inQuery) + if err != nil { + logrus.Errorf("第 %d 页请求失败: %v", page, err) + recordFailure(ctx, platform.PlatformCode, iface.Code, taskType, fmt.Sprintf("第 %d 页请求失败: %v", page, err)) + continue + } + rows, _, mt, _, pe := parseRespExt(resp.Body, iface.ResponseConfig) + if pe != nil { + logrus.Errorf("第 %d 页解析失败: %v", page, pe) + recordFailure(ctx, platform.PlatformCode, iface.Code, taskType, fmt.Sprintf("第 %d 页解析失败: %v", page, pe)) + continue + } + injectRowFields(rows, body, iface.RequestConfig) + inserted, _ = savePage(ctx, td, rows) + result.InsertedRows += inserted + result.TotalRows += len(rows) + if mt > maxTime { + maxTime = mt + } + time.Sleep(100 * time.Millisecond) + } + } } else { - // 普通分页 + // 普通分页（无 response_config） for page := 2; page <= totalPages; page++ { body := buildReqBody(iface, page, pageSize, lastSyncTime, nil) - resp, err := api.Request(ctx, method, iface.Url, body, inQuery) + resp, err = api.Request(ctx, method, iface.Url, body, inQuery) if err != nil { logrus.Errorf("第 %d 页请求失败: %v", page, err) - recordFailure(ctx, platform.PlatformCode, iface.Code, taskType, fmt.Sprintf("第 %d 页请求失败: %v", page, err)) continue } rows, _, mt, _, pe := parseRespExt(resp.Body, iface.ResponseConfig) if pe != nil { logrus.Errorf("第 %d 页解析失败: %v", page, pe) - recordFailure(ctx, platform.PlatformCode, iface.Code, taskType, fmt.Sprintf("第 %d 页解析失败: %v", page, pe)) continue } + injectRowFields(rows, body, iface.RequestConfig) inserted, _ = savePage(ctx, td, rows) result.InsertedRows += inserted result.TotalRows += len(rows) @@ -238,6 +316,33 @@ func isCursorPagination(iface *entity.ApiInterface) bool { return cp } +// hasMoreCheck 从响应体中提取 has_more_field 的值 +func hasMoreCheck(raw []byte, hasMorePath string) bool { + var respMap map[string]interface{} + if err := json.Unmarshal(raw, &respMap); err != nil { + return false + } + parts := strings.Split(hasMorePath, ".") + cc := respMap + for i, p := range parts { + if i == len(parts)-1 { + if b, ok := cc[p].(bool); ok { + return b + } + if s, ok := cc[p].(string); ok { + return s == "true" + } + return false + } + if m, ok := cc[p].(map[string]interface{}); ok { + cc = m + } else { + return false + } + } + return false +} + // collectPrefetchEntities 从 rows 中收集实体和行数据 func collectPrefetchEntities(rows []map[string]interface{}, prefetch *PrefetchConfig, allEntities *[]interface{}, allRows *[]map[string]interface{}) { for _, item := range rows { @@ -266,6 +371,12 @@ func syncWithPrefetch(ctx context.Context, api *ApiClient, platform *PlatformCon // ====== 1. 预取阶段：分页拉取全部实体列表 ====== prefetchIface := findInterfaceByURL(allIfaces, prefetch.URL) + // 判断预取来源是否有递归配置（如钉钉部门树） + var prefetchRecursiveCfg *RecursiveConfig + if prefetchIface != nil { + prefetchRecursiveCfg = parseRecursiveConfig(prefetchIface.RequestConfig) + } + // 判断预取来源是否游标分页，以及分页参数名 prefetchIsCursor := false prefetchPageParam := "page" @@ -303,69 +414,127 @@ func syncWithPrefetch(ctx context.Context, api *ApiClient, platform *PlatformCon allEntities := make([]interface{}, 0) allRows := make([]map[string]interface{}, 0) - // 第一页（游标分页首次 cursor=""） - firstExtra := make(map[string]interface{}) - if prefetchIsCursor { - firstExtra[prefetchPageParam] = "" - } prefetchReqIface := prefetchIface if prefetchReqIface == nil { prefetchReqIface = iface } - body := buildReqBody(prefetchReqIface, 1, prefetchPageSize, lastSyncTime, firstExtra) - resp, err := api.Request(ctx, prefetchMethod, prefetch.URL, body, prefetchInQuery) - if err != nil { - recordFailure(ctx, platform.PlatformCode, iface.Code, taskType, fmt.Sprintf("预取第一页请求失败: %v", err)) - return nil, fmt.Errorf("预取第一页失败: %w", err) - } - rows, prefetchTotalPages, _, nextCursor, err := parseRespExt(resp.Body, prefetchRespCfg) - if err != nil { - recordFailure(ctx, platform.PlatformCode, iface.Code, taskType, fmt.Sprintf("解析预取响应失败: %v", err)) - return nil, fmt.Errorf("解析预取响应失败: %w", err) - } - collectPrefetchEntities(rows, prefetch, &allEntities, &allRows) + if prefetchIface != nil && prefetchRecursiveCfg != nil { + // ----- 递归遍历预取（如钉钉部门树）----- + maxDepth := 20 + if md, ok := prefetchIface.RequestConfig["max_recursive_depth"].(float64); ok { + maxDepth = int(md) + } + processedKeys := make(map[string]bool) + type rItem struct { + depth int + keyVal interface{} + } + queue := []rItem{{depth: 0, keyVal: nil}} - // 分页循环 - if prefetchIsCursor { - // 游标分页 - for nextCursor != "" && nextCursor != "nomore" { - body := buildReqBody(prefetchReqIface, 1, prefetchPageSize, lastSyncTime, map[string]interface{}{ - prefetchPageParam: nextCursor, - }) - resp, err := api.Request(ctx, prefetchMethod, prefetch.URL, body, prefetchInQuery) + for len(queue) > 0 { + item := queue[0] + queue = queue[1:] + if item.depth > maxDepth { + continue + } + if item.keyVal != nil { + keyStr := fmt.Sprintf("%v", item.keyVal) + if processedKeys[keyStr] { + continue + } + processedKeys[keyStr] = true + } + extra := make(map[string]interface{}) + if item.keyVal != nil { + extra[prefetchRecursiveCfg.TargetParam] = item.keyVal + } + body := buildReqBody(prefetchReqIface, 1, prefetchPageSize, 0, extra) + r2, err := api.Request(ctx, prefetchMethod, prefetch.URL, body, prefetchInQuery) if err != nil { - logrus.Errorf("预取游标 %s 请求失败: %v", nextCursor, err) - break + logrus.Errorf("预取递归 [depth=%d] 请求失败: %v", item.depth, err) + continue } - rows, _, _, nc, pe := parseRespExt(resp.Body, prefetchRespCfg) + itemRows, _, _, _, pe := parseRespExt(r2.Body, prefetchRespCfg) if pe != nil { - logrus.Errorf("预取游标 %s 解析失败: %v", nextCursor, pe) - break + logrus.Errorf("预取递归 [depth=%d] 解析失败: %v", item.depth, pe) + continue } - if len(rows) == 0 { - break + for _, row := range itemRows { + allRows = append(allRows, row) + if prefetch.ValueField == "" { + allEntities = append(allEntities, row) + } else if v, ok := row[prefetch.ValueField]; ok { + if f, ok := v.(float64); ok { + allEntities = append(allEntities, int64(f)) + } else { + allEntities = append(allEntities, v) + } + } + if v, ok := row[prefetchRecursiveCfg.KeyField]; ok { + queue = append(queue, rItem{depth: item.depth + 1, keyVal: v}) + } } - nextCursor = nc - collectPrefetchEntities(rows, prefetch, &allEntities, &allRows) time.Sleep(100 * time.Millisecond) } } else { - // 普通分页 - for page := 2; page <= prefetchTotalPages; page++ { - body := buildReqBody(prefetchReqIface, page, prefetchPageSize, lastSyncTime, nil) - resp, err := api.Request(ctx, prefetchMethod, prefetch.URL, body, prefetchInQuery) - if err != nil { - logrus.Errorf("预取第 %d 页请求失败: %v", page, err) - continue + // ----- 常规分页预取 ----- + firstExtra := make(map[string]interface{}) + if prefetchIsCursor { + firstExtra[prefetchPageParam] = "" + } + body := buildReqBody(prefetchReqIface, 1, prefetchPageSize, lastSyncTime, firstExtra) + resp, err := api.Request(ctx, prefetchMethod, prefetch.URL, body, prefetchInQuery) + if err != nil { + recordFailure(ctx, platform.PlatformCode, iface.Code, taskType, fmt.Sprintf("预取第一页请求失败: %v", err)) + return nil, fmt.Errorf("预取第一页失败: %w", err) + } + + rows, prefetchTotalPages, _, nextCursor, err := parseRespExt(resp.Body, prefetchRespCfg) + if err != nil { + recordFailure(ctx, platform.PlatformCode, iface.Code, taskType, fmt.Sprintf("解析预取响应失败: %v", err)) + return nil, fmt.Errorf("解析预取响应失败: %w", err) + } + collectPrefetchEntities(rows, prefetch, &allEntities, &allRows) + + if prefetchIsCursor { + for nextCursor != "" && nextCursor != "nomore" { + body := buildReqBody(prefetchReqIface, 1, prefetchPageSize, lastSyncTime, map[string]interface{}{ + prefetchPageParam: nextCursor, + }) + resp, err := api.Request(ctx, prefetchMethod, prefetch.URL, body, prefetchInQuery) + if err != nil { + logrus.Errorf("预取游标 %s 请求失败: %v", nextCursor, err) + break + } + rows, _, _, nc, pe := parseRespExt(resp.Body, prefetchRespCfg) + if pe != nil { + logrus.Errorf("预取游标 %s 解析失败: %v", nextCursor, pe) + break + } + if len(rows) == 0 { + break + } + nextCursor = nc + collectPrefetchEntities(rows, prefetch, &allEntities, &allRows) + time.Sleep(100 * time.Millisecond) } - rows, _, _, _, pe := parseRespExt(resp.Body, prefetchRespCfg) - if pe != nil { - logrus.Errorf("预取第 %d 页解析失败: %v", page, pe) - continue + } else { + for page := 2; page <= prefetchTotalPages; page++ { + body := buildReqBody(prefetchReqIface, page, prefetchPageSize, lastSyncTime, nil) + resp, err := api.Request(ctx, prefetchMethod, prefetch.URL, body, prefetchInQuery) + if err != nil { + logrus.Errorf("预取第 %d 页请求失败: %v", page, err) + continue + } + rows, _, _, _, pe := parseRespExt(resp.Body, prefetchRespCfg) + if pe != nil { + logrus.Errorf("预取第 %d 页解析失败: %v", page, pe) + continue + } + collectPrefetchEntities(rows, prefetch, &allEntities, &allRows) + time.Sleep(100 * time.Millisecond) } - collectPrefetchEntities(rows, prefetch, &allEntities, &allRows) - time.Sleep(100 * time.Millisecond) } } @@ -375,7 +544,7 @@ func syncWithPrefetch(ctx context.Context, api *ApiClient, platform *PlatformCon } logrus.Infof("预取到 %d 个实体", len(allEntities)) - // 2. 将预取的数据也存入库（如账户列表存入 tencent_account_relation） + // 将预取的数据也存入库（如账户列表存入 tencent_account_relation） if prefetchIface != nil && prefetchIface.TableDefinition != nil { prefetchTd, err := ParseTableDefinition(prefetchIface.TableDefinition) if err == nil { @@ -386,11 +555,13 @@ func syncWithPrefetch(ctx context.Context, api *ApiClient, platform *PlatformCon } } - // 2. 并发处理每个实体的数据 + // 并发处理每个实体的数据 result := &SyncResult{TableName: td.TableName} pageSize := GetSyncPageSize(ctx) if ps, ok := iface.RequestConfig["page_size"].(float64); ok { pageSize = int(ps) + } else if ps, ok := iface.RequestConfig["pageSize"].(float64); ok { + pageSize = int(ps) } dataMethod := string(iface.Method) @@ -411,52 +582,118 @@ func syncWithPrefetch(ctx context.Context, api *ApiClient, platform *PlatformCon defer func() { <-sem }() logrus.Infof(" 处理实体 [%d/%d]: %v", idx+1, len(allEntities), val) - - page := 1 - totalPages := 1 entityMaxTime := int64(0) - for page <= totalPages { - body := buildReqBody(iface, page, pageSize, lastSyncTime, map[string]interface{}{ + if isCursorPagination(iface) { + // ----- 游标分页（如钉钉 user_list）----- + cp := "cursor" + if p, ok := iface.RequestConfig["page_param"].(string); ok && p != "" { + cp = p + } + firstExtra := map[string]interface{}{ prefetch.TargetParam: val, - }) - + } + if icv, ok := iface.RequestConfig["initial_cursor"]; ok { + firstExtra[cp] = icv + } else { + firstExtra[cp] = "" + } + body := buildReqBody(iface, 1, pageSize, lastSyncTime, firstExtra) resp, err := api.Request(ctx, dataMethod, iface.Url, body, inQuery) if err != nil { - logrus.Errorf(" 实体 %v 第 %d 页失败: %v", val, page, err) - page++ - time.Sleep(200 * time.Millisecond) - continue + logrus.Errorf(" 实体 %v 首次请求失败: %v", val, err) + return } - - rows, tp, mt, parseErr := parseResp(resp.Body, iface.ResponseConfig) - if parseErr != nil { - logrus.Errorf(" 解析响应失败: %v", parseErr) - page++ - continue + rows, _, mt, nc, pe := parseRespExt(resp.Body, iface.ResponseConfig) + if pe != nil { + logrus.Errorf(" 实体 %v 解析首页失败: %v", val, pe) + return } - - if page == 1 { - totalPages = tp - } - for i := range rows { rows[i][prefetch.TargetParam] = val } - + injectRowFields(rows, body, iface.RequestConfig) inserted, _ := savePage(ctx, td, rows) - mu.Lock() result.InsertedRows += inserted result.TotalRows += len(rows) mu.Unlock() - if mt > entityMaxTime { entityMaxTime = mt } - - page++ - time.Sleep(100 * time.Millisecond) + nextCursor := nc + for nextCursor != "" && nextCursor != "nomore" { + body := buildReqBody(iface, 1, pageSize, lastSyncTime, map[string]interface{}{ + cp: nextCursor, + prefetch.TargetParam: val, + }) + resp, err := api.Request(ctx, dataMethod, iface.Url, body, inQuery) + if err != nil { + logrus.Errorf(" 实体 %v 游标 %s 失败: %v", val, nextCursor, err) + break + } + rows, _, mt, nc, pe := parseRespExt(resp.Body, iface.ResponseConfig) + if pe != nil { + logrus.Errorf(" 实体 %v 游标 %s 解析失败: %v", val, nextCursor, pe) + break + } + if len(rows) == 0 { + break + } + nextCursor = nc + for i := range rows { + rows[i][prefetch.TargetParam] = val + } + injectRowFields(rows, body, iface.RequestConfig) + inserted, _ := savePage(ctx, td, rows) + mu.Lock() + result.InsertedRows += inserted + result.TotalRows += len(rows) + mu.Unlock() + if mt > entityMaxTime { + entityMaxTime = mt + } + time.Sleep(100 * time.Millisecond) + } + } else { + // ----- 普通分页 ----- + page := 1 + totalPages := 1 + for page <= totalPages { + body := buildReqBody(iface, page, pageSize, lastSyncTime, map[string]interface{}{ + prefetch.TargetParam: val, + }) + resp, err := api.Request(ctx, dataMethod, iface.Url, body, inQuery) + if err != nil { + logrus.Errorf(" 实体 %v 第 %d 页失败: %v", val, page, err) + page++ + time.Sleep(200 * time.Millisecond) + continue + } + rows, tp, mt, parseErr := parseResp(resp.Body, iface.ResponseConfig) + if parseErr != nil { + logrus.Errorf(" 解析响应失败: %v", parseErr) + page++ + continue + } + if page == 1 { + totalPages = tp + } + for i := range rows { + rows[i][prefetch.TargetParam] = val + } + injectRowFields(rows, body, iface.RequestConfig) + inserted, _ := savePage(ctx, td, rows) + mu.Lock() + result.InsertedRows += inserted + result.TotalRows += len(rows) + mu.Unlock() + if mt > entityMaxTime { + entityMaxTime = mt + } + page++ + time.Sleep(100 * time.Millisecond) + } } if entityMaxTime > 0 { @@ -481,6 +718,90 @@ func syncWithPrefetch(ctx context.Context, api *ApiClient, platform *PlatformCon return result, nil } +// syncRecursive 递归遍历同步（如钉钉部门树：先查根级 → 对每个子部门递归查下级） +func syncRecursive(ctx context.Context, api *ApiClient, platform *PlatformConfig, iface *entity.ApiInterface, td *TableDefinition, recursive *RecursiveConfig, start time.Time) (*SyncResult, error) { + maxDepth := 20 + if md, ok := iface.RequestConfig["max_recursive_depth"].(float64); ok { + maxDepth = int(md) + } + + inQuery := paramsInQuery(iface) + method := string(iface.Method) + + allRows := make([]map[string]interface{}, 0) + processedKeys := make(map[string]bool) + + type queueItem struct { + depth int + keyVal interface{} // nil 表示根级 + } + queue := []queueItem{{depth: 0, keyVal: nil}} + + for len(queue) > 0 { + item := queue[0] + queue = queue[1:] + + if item.depth > maxDepth { + logrus.Warnf("递归已达最大深度 %d，终止该分支", maxDepth) + continue + } + + // 防重复处理 + if item.keyVal != nil { + keyStr := fmt.Sprintf("%v", item.keyVal) + if processedKeys[keyStr] { + continue + } + processedKeys[keyStr] = true + } + + extraParams := make(map[string]interface{}) + if item.keyVal != nil { + extraParams[recursive.TargetParam] = item.keyVal + } + + body := buildReqBody(iface, 1, 100, 0, extraParams) + resp, err := api.Request(ctx, method, iface.Url, body, inQuery) + if err != nil { + logrus.Errorf("递归 [depth=%d] 请求失败: %v", item.depth, err) + recordFailure(ctx, platform.PlatformCode, iface.Code, "full", fmt.Sprintf("递归深度 %d 请求失败: %v", item.depth, err)) + continue + } + + rows, _, _, _, err := parseRespExt(resp.Body, iface.ResponseConfig) + if err != nil { + logrus.Errorf("递归 [depth=%d] 解析失败: %v", item.depth, err) + continue + } + + for _, row := range rows { + allRows = append(allRows, row) + if v, ok := row[recursive.KeyField]; ok { + queue = append(queue, queueItem{depth: item.depth + 1, keyVal: v}) + } + } + + time.Sleep(100 * time.Millisecond) + } + + if len(allRows) == 0 { + logrus.Warn("递归结果为空，跳过入库") + return &SyncResult{TableName: td.TableName, Duration: fmt.Sprintf("%.1fs", time.Since(start).Seconds())}, nil + } + + inserted, _ := savePage(ctx, td, allRows) + updateSyncTime(ctx, platform.PlatformCode, iface.Code, time.Now().Unix()) + + result := &SyncResult{ + TableName: td.TableName, + TotalRows: len(allRows), + InsertedRows: inserted, + Duration: fmt.Sprintf("%.1fs", time.Since(start).Seconds()), + } + logrus.Infof("递归同步完成 - 表:%s, %d条, 写入%d条, 耗时%s", td.TableName, result.TotalRows, result.InsertedRows, result.Duration) + return result, nil +} + // getTotalPages 从响应中提取总页数 func getTotalPages(raw []byte) int { rows, tp, _, _, err := parseRespExt(raw, nil) @@ -498,6 +819,12 @@ func toFloat64(v interface{}) (float64, bool) { return float64(val), true case int64: return float64(val), true + case string: + // 支持字符串类型的成功值（如钉钉智能薪酬返回 code: "200"） + if f, err := strconv.ParseFloat(val, 64); err == nil { + return f, true + } + return 0, false default: return 0, false } @@ -524,7 +851,10 @@ func buildPrefetchParams(iface *entity.ApiInterface) map[string]interface{} { k == "page_size_param" || k == "time_field" || k == "parameters_location" || k == "filtering" || k == "group_by" || k == "date_range" || k == "body_wrapper_field" || k == "exclude_from_wrapper" || - k == "cursor_pagination" || k == "time_field_mode" { + k == "cursor_pagination" || k == "time_field_mode" || + k == "recursive" || k == "max_recursive_depth" || + k == "initial_cursor" || k == "pagination_mode" || + k == "full_sync_start_time" || k == "row_inject" { continue } if k == pageParam || k == psParam { @@ -567,6 +897,33 @@ func parsePrefetchConfig(requestConfig map[string]interface{}) *PrefetchConfig { return pc } +// parseRecursiveConfig 解析递归遍历配置 +func parseRecursiveConfig(requestConfig map[string]interface{}) *RecursiveConfig { + if requestConfig == nil { + return nil + } + raw, ok := requestConfig["recursive"] + if !ok || raw == nil { + return nil + } + m, ok := raw.(map[string]interface{}) + if !ok { + return nil + } + rc := &RecursiveConfig{} + if kf, _ := m["key_field"].(string); kf != "" { + rc.KeyField = kf + } else { + return nil + } + if tp, _ := m["target_param"].(string); tp != "" { + rc.TargetParam = tp + } else { + return nil + } + return rc +} + // extractValues 从 JSON 响应中提取值列表 func extractValues(raw []byte, path, valueField string) ([]interface{}, error) { var resp map[string]interface{} @@ -612,7 +969,10 @@ func buildReqBody(iface *entity.ApiInterface, page, pageSize int, lastSyncTime i k == "page_param" || k == "page_size_param" || k == "parameters_location" || k == "cursor_pagination" || k == "time_field_mode" || k == "body_wrapper_field" || k == "exclude_from_wrapper" || - k == "top_level_params" { + k == "top_level_params" || k == "recursive" || + k == "max_recursive_depth" || k == "initial_cursor" || + k == "pagination_mode" || k == "full_sync_start_time" || + k == "row_inject" { continue } body[k] = v @@ -628,39 +988,68 @@ func buildReqBody(iface *entity.ApiInterface, page, pageSize int, lastSyncTime i psParam = p } } - body[pageParam] = page + // 偏移量分页（如钉钉 offset）：offset = (page-1) * pageSize + paginationMode := "" + if iface.RequestConfig != nil { + if pm, ok := iface.RequestConfig["pagination_mode"].(string); ok { + paginationMode = pm + } + } + if paginationMode == "offset" { + body[pageParam] = (page - 1) * pageSize + } else { + body[pageParam] = page + } body[psParam] = pageSize // 时间过滤处理：支持两种模式 // 1. "filtering" 模式（默认）：生成 filtering=[{"field":"...","operator":"GREATER_EQUALS","values":["..."]}]（腾讯） // 2. "range" 模式：生成 beginTime/endTime + queryType（快手） - if tf, ok := iface.RequestConfig["time_field"].(string); ok && tf != "" { - timeMode := "filtering" - if tm, ok := iface.RequestConfig["time_field_mode"].(string); ok && tm != "" { - timeMode = tm - } + if iface.RequestConfig != nil { + if tf, ok := iface.RequestConfig["time_field"].(string); ok && tf != "" { + timeMode := "filtering" + if tm, ok := iface.RequestConfig["time_field_mode"].(string); ok && tm != "" { + timeMode = tm + } - if timeMode == "range" { - // 快手模式：beginTime/endTime（毫秒时间戳） - timeMs := lastSyncTime - if timeMs <= 0 { - // 全量：默认90天前 - timeMs = time.Now().Add(-90 * 24 * time.Hour).UnixMilli() - } - body["queryType"] = 2 - body["beginTime"] = timeMs - body["endTime"] = time.Now().UnixMilli() - } else if lastSyncTime > 0 { - // 腾讯 filtering 模式（仅增量时） - timeFilter := map[string]interface{}{ - "field": tf, - "operator": "GREATER_EQUALS", - "values": []interface{}{fmt.Sprintf("%d", lastSyncTime)}, - } - if existing, ok := body["filtering"].([]interface{}); ok { - body["filtering"] = append(existing, timeFilter) - } else { - body["filtering"] = []interface{}{timeFilter} + if timeMode == "range" { + // 快手模式：beginTime/endTime（毫秒时间戳） + timeMs := lastSyncTime + if timeMs <= 0 { + // 全量：优先使用配置的 full_sync_start_time，否则默认90天前 + if fst, ok := iface.RequestConfig["full_sync_start_time"].(float64); ok && fst > 0 { + timeMs = int64(fst) + } else { + timeMs = time.Now().Add(-90 * 24 * time.Hour).UnixMilli() + } + } + body["queryType"] = 2 + body["beginTime"] = timeMs + body["endTime"] = time.Now().UnixMilli() + } else if lastSyncTime > 0 { + // 腾讯 filtering 模式（仅增量时） + timeFilter := map[string]interface{}{ + "field": tf, + "operator": "GREATER_EQUALS", + "values": []interface{}{fmt.Sprintf("%d", lastSyncTime)}, + } + if existing, ok := body["filtering"].([]interface{}); ok { + body["filtering"] = append(existing, timeFilter) + } else { + body["filtering"] = []interface{}{timeFilter} + } + } else if fst, ok := iface.RequestConfig["full_sync_start_time"].(float64); ok && fst > 0 { + // 全量 filtering 模式：指定了 full_sync_start_time，从该时间戳开始拉取 + timeFilter := map[string]interface{}{ + "field": tf, + "operator": "GREATER_EQUALS", + "values": []interface{}{fmt.Sprintf("%d", int64(fst))}, + } + if existing, ok := body["filtering"].([]interface{}); ok { + body["filtering"] = append(existing, timeFilter) + } else { + body["filtering"] = []interface{}{timeFilter} + } } } } @@ -687,8 +1076,12 @@ func buildReqBody(iface *entity.ApiInterface, page, pageSize int, lastSyncTime i delete(body, k) } } - b, _ := json.Marshal(wrapperObj) - body[wf] = string(b) + b, err := json.Marshal(wrapperObj) + if err != nil { + logrus.Errorf("JSON序列化 wrapper 失败: %v", err) + } else { + body[wf] = string(b) + } } } @@ -703,6 +1096,7 @@ func parseRespExt(raw []byte, rc map[string]interface{}) ([]map[string]interface } successField, successVal := "code", float64(0) msgField, listPath, cursorPath := "message", "data", "" + hasMorePath := "" singleRecord := false if rc != nil { if sf, _ := rc["success_field"].(string); sf != "" { @@ -725,6 +1119,9 @@ func parseRespExt(raw []byte, rc map[string]interface{}) ([]map[string]interface if sr, _ := rc["single_record"].(bool); sr { singleRecord = true } + if hm, _ := rc["has_more_field"].(string); hm != "" { + hasMorePath = hm + } } if v, ok := respMap[successField]; ok { actual, _ := toFloat64(v) @@ -820,6 +1217,23 @@ func parseRespExt(raw []byte, rc map[string]interface{}) ([]map[string]interface if i == len(cp)-1 { if s, ok := cc[p].(string); ok { nextCursor = s + } else if f, ok := cc[p].(float64); ok { + // 数字游标（如钉钉 next_cursor=10） + nextCursor = fmt.Sprintf("%.0f", f) + } + } else if m, ok := cc[p].(map[string]interface{}); ok { + cc = m + } + } + } + // has_more 字段支持：false 时标记游标结束 + if hasMorePath != "" { + parts := strings.Split(hasMorePath, ".") + cc := respMap + for i, p := range parts { + if i == len(parts)-1 { + if b, ok := cc[p].(bool); ok && !b { + nextCursor = "nomore" } } else if m, ok := cc[p].(map[string]interface{}); ok { cc = m @@ -950,3 +1364,32 @@ func findInterfaceByURL(ifaces []entity.ApiInterface, url string) *entity.ApiInt } return nil } + +// injectRowFields 将请求参数中 row_inject 指定的字段注入到响应行中 +// 用于需要将请求参数（如 statisticsMonth）持久化到表中，但响应不含该字段的场景 +func injectRowFields(rows []map[string]interface{}, body map[string]interface{}, requestConfig map[string]interface{}) { + if requestConfig == nil || body == nil { + return + } + rawInject, ok := requestConfig["row_inject"] + if !ok { + return + } + injectList, ok := rawInject.([]interface{}) + if !ok { + return + } + for _, item := range injectList { + fieldName, ok := item.(string) + if !ok { + continue + } + val, exists := body[fieldName] + if !exists { + continue + } + for i := range rows { + rows[i][fieldName] = val + } + } +} diff --git a/service/sync/helpers.go b/service/sync/helpers.go index 1c49a1b..27c98d4 100644 --- a/service/sync/helpers.go +++ b/service/sync/helpers.go @@ -41,3 +41,12 @@ func GetRetryCount(ctx context.Context) int { } return r } + +// GetSyncTimeout 获取单次同步超时时间（分钟，默认120），全量超大表可适当调大 +func GetSyncTimeout(ctx context.Context) int { + t := g.Cfg().MustGet(ctx, "sync.sync_timeout_minutes", 120).Int() + if t < 1 { + return 120 + } + return t +} diff --git a/service/sync/platform_manager.go b/service/sync/platform_manager.go index 8f362f7..bd89483 100644 --- a/service/sync/platform_manager.go +++ b/service/sync/platform_manager.go @@ -63,6 +63,15 @@ func (m *PlatformManager) GetPlatform(ctx context.Context, platformCode string) cfg.AppSecret = as } } + case "APP_SIGNATURE": + if platform.AuthConfig != nil { + if aid, _ := platform.AuthConfig["app_id"].(string); aid != "" { + cfg.AppKey = aid + } + if as, _ := platform.AuthConfig["app_secret"].(string); as != "" { + cfg.AppSecret = as + } + } default: logrus.Warnf("平台 %s 认证类型 %s 未处理", platformCode, platform.AuthType) } diff --git a/service/sync/sync_scheduler.go b/service/sync/sync_scheduler.go index be9890c..8ed8fd7 100644 --- a/service/sync/sync_scheduler.go +++ b/service/sync/sync_scheduler.go @@ -7,7 +7,7 @@ import ( dao "dataengine/dao/dict" dto "dataengine/model/dto/dict" - "gitea.com/red-future/common/beans" + "gitea.redpowerfuture.com/red-future/common/beans" "github.com/gogf/gf/v2/frame/g" "github.com/sirupsen/logrus" ) @@ -71,7 +71,7 @@ func runAutoSync(ctx context.Context) { // InitAndStartAutoSync 在 main 中调用：初始化配置后启动自动同步和补偿 func InitAndStartAutoSync(ctx context.Context) { // 读取配置中的同步开关 - enabled := g.Cfg().MustGet(ctx, "sync.auto_sync_enabled", true).Bool() + enabled := g.Cfg().MustGet(ctx, "sync.auto_sync_enabled", false).Bool() if enabled { go StartAutoSync(ctx) } else { diff --git a/service/sync/table_manager.go b/service/sync/table_manager.go index d28c3e7..0f1450b 100644 --- a/service/sync/table_manager.go +++ b/service/sync/table_manager.go @@ -5,7 +5,7 @@ import ( "fmt" "strings" - "gitea.com/red-future/common/db/gfdb" + "gitea.redpowerfuture.com/red-future/common/db/gfdb" "github.com/sirupsen/logrus" ) diff --git a/sql/report_add_tenant_id.sql b/sql/report_add_tenant_id.sql new file mode 100644 index 0000000..6b8eef2 --- /dev/null +++ b/sql/report_add_tenant_id.sql @@ -0,0 +1,117 @@ +-- ============================================================ +-- 报表系统表 - 新增 tenant_id 迁移脚本 +-- 为已存在的5张报表系统表添加租户隔离字段 +-- 执行前请确认当前数据库中表是否存在 +-- ============================================================ + +-- ---------------------------------------------------------- +-- 1. report_business_config (业务配置表) +-- ---------------------------------------------------------- + +-- 添加 tenant_id 列 +ALTER TABLE IF EXISTS report_business_config + ADD COLUMN IF NOT EXISTS tenant_id BIGINT NOT NULL DEFAULT 0; + +-- 重建唯一索引（加入 tenant_id） +DROP INDEX IF EXISTS idx_business_code; +CREATE UNIQUE INDEX IF NOT EXISTS idx_business_code + ON report_business_config (tenant_id, business_code); + +COMMENT ON COLUMN report_business_config.tenant_id IS '租户ID'; + + +-- ---------------------------------------------------------- +-- 2. report_report_config (报表配置表) +-- ---------------------------------------------------------- + +-- 添加 tenant_id 列 +ALTER TABLE IF EXISTS report_report_config + ADD COLUMN IF NOT EXISTS tenant_id BIGINT NOT NULL DEFAULT 0; + +-- 重建唯一约束（加入 tenant_id） +ALTER TABLE IF EXISTS report_report_config + DROP CONSTRAINT IF EXISTS uk_business_report_code; +ALTER TABLE IF EXISTS report_report_config + ADD CONSTRAINT uk_business_report_code UNIQUE (tenant_id, business_code, report_code); + +-- 重建索引 +DROP INDEX IF EXISTS idx_report_business_code; +CREATE INDEX IF NOT EXISTS idx_report_business_code + ON report_report_config (tenant_id, business_code); + +COMMENT ON COLUMN report_report_config.tenant_id IS '租户ID'; + + +-- ---------------------------------------------------------- +-- 3. report_field_config (字段配置表) +-- ---------------------------------------------------------- + +-- 添加 tenant_id 列 +ALTER TABLE IF EXISTS report_field_config + ADD COLUMN IF NOT EXISTS tenant_id BIGINT NOT NULL DEFAULT 0; + +-- 重建唯一约束（加入 tenant_id） +ALTER TABLE IF EXISTS report_field_config + DROP CONSTRAINT IF EXISTS uk_business_report_field_code; +ALTER TABLE IF EXISTS report_field_config + ADD CONSTRAINT uk_business_report_field_code UNIQUE (tenant_id, business_code, report_code, field_code); + +-- 重建索引 +DROP INDEX IF EXISTS idx_field_business_report; +CREATE INDEX IF NOT EXISTS idx_field_business_report + ON report_field_config (tenant_id, business_code, report_code); +DROP INDEX IF EXISTS idx_field_data_type; +CREATE INDEX IF NOT EXISTS idx_field_data_type + ON report_field_config (tenant_id, data_type); +DROP INDEX IF EXISTS idx_field_field_role; +CREATE INDEX IF NOT EXISTS idx_field_field_role + ON report_field_config (tenant_id, field_role); + +COMMENT ON COLUMN report_field_config.tenant_id IS '租户ID'; + + +-- ---------------------------------------------------------- +-- 4. report_extract_config (抽取配置表) +-- ---------------------------------------------------------- + +-- 添加 tenant_id 列 +ALTER TABLE IF EXISTS report_extract_config + ADD COLUMN IF NOT EXISTS tenant_id BIGINT NOT NULL DEFAULT 0; + +-- 重建唯一约束（加入 tenant_id） +ALTER TABLE IF EXISTS report_extract_config + DROP CONSTRAINT IF EXISTS uk_business_report_extract_code; +ALTER TABLE IF EXISTS report_extract_config + ADD CONSTRAINT uk_business_report_extract_code UNIQUE (tenant_id, business_code, report_code, extract_code); + +-- 重建索引 +DROP INDEX IF EXISTS idx_extract_business_report; +CREATE INDEX IF NOT EXISTS idx_extract_business_report + ON report_extract_config (tenant_id, business_code, report_code); + +COMMENT ON COLUMN report_extract_config.tenant_id IS '租户ID'; + + +-- ---------------------------------------------------------- +-- 5. report_extract_log (抽取记录表) +-- ---------------------------------------------------------- + +-- 添加 tenant_id 列 +ALTER TABLE IF EXISTS report_extract_log + ADD COLUMN IF NOT EXISTS tenant_id BIGINT NOT NULL DEFAULT 0; + +-- 重建唯一约束（加入 tenant_id） +ALTER TABLE IF EXISTS report_extract_log + DROP CONSTRAINT IF EXISTS uk_extract_keys; +ALTER TABLE IF EXISTS report_extract_log + ADD CONSTRAINT uk_extract_keys UNIQUE (tenant_id, business_code, report_code, extract_code, stat_date); + +-- 重建索引 +DROP INDEX IF EXISTS idx_extract_log_business_report; +CREATE INDEX IF NOT EXISTS idx_extract_log_business_report + ON report_extract_log (tenant_id, business_code, report_code); +DROP INDEX IF EXISTS idx_extract_log_stat_date; +CREATE INDEX IF NOT EXISTS idx_extract_log_stat_date + ON report_extract_log (tenant_id, stat_date); + +COMMENT ON COLUMN report_extract_log.tenant_id IS '租户ID'; diff --git a/sql/report_common_ddl.sql b/sql/report_common_ddl.sql new file mode 100644 index 0000000..29a8ecf --- /dev/null +++ b/sql/report_common_ddl.sql @@ -0,0 +1,301 @@ +-- ============================================================ +-- 通用报表公共包 - 系统表 DDL +-- 业务隔离、多业务支持、表/字段 100% 配置化 +-- ============================================================ + +-- ---------------------------------------------------------- +-- 1. 业务配置表 (business_config) +-- ---------------------------------------------------------- +CREATE TABLE IF NOT EXISTS report_business_config ( + id BIGSERIAL PRIMARY KEY, + tenant_id BIGINT NOT NULL DEFAULT 0, + business_code VARCHAR(64) NOT NULL, + business_name VARCHAR(128) NOT NULL, + description TEXT DEFAULT '', + status VARCHAR(16) NOT NULL DEFAULT 'ACTIVE', + config JSONB DEFAULT '{}', + creator VARCHAR(64) DEFAULT '', + created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(), + updater VARCHAR(64) DEFAULT '', + updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(), + deleted_at TIMESTAMP WITH TIME ZONE +); + +COMMENT ON TABLE report_business_config IS '报表业务配置表'; +COMMENT ON COLUMN report_business_config.tenant_id IS '租户ID'; +COMMENT ON COLUMN report_business_config.business_code IS '业务编码(唯一标识)'; +COMMENT ON COLUMN report_business_config.business_name IS '业务名称'; +COMMENT ON COLUMN report_business_config.description IS '业务描述'; +COMMENT ON COLUMN report_business_config.status IS '状态: ACTIVE启用/INACTIVE停用'; +COMMENT ON COLUMN report_business_config.config IS '业务级配置(JSON)，如默认数据源等'; + +CREATE UNIQUE INDEX IF NOT EXISTS idx_business_code ON report_business_config (tenant_id, business_code); + +-- ---------------------------------------------------------- +-- 2. 报表配置表 (report_config) +-- ---------------------------------------------------------- +CREATE TABLE IF NOT EXISTS report_report_config ( + id BIGSERIAL PRIMARY KEY, + tenant_id BIGINT NOT NULL DEFAULT 0, + business_code VARCHAR(64) NOT NULL, + report_code VARCHAR(64) NOT NULL, + report_name VARCHAR(128) NOT NULL, + description TEXT DEFAULT '', + status VARCHAR(16) NOT NULL DEFAULT 'ACTIVE', + stat_table_name VARCHAR(128) NOT NULL, + stat_table_comment VARCHAR(256) DEFAULT '', + date_field VARCHAR(64) DEFAULT 'stat_date', + primary_keys JSONB DEFAULT '["id"]'::jsonb, + conflict_keys JSONB DEFAULT '["stat_date"]'::jsonb, + config JSONB DEFAULT '{}', + creator VARCHAR(64) DEFAULT '', + created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(), + updater VARCHAR(64) DEFAULT '', + updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(), + deleted_at TIMESTAMP WITH TIME ZONE, + + CONSTRAINT uk_business_report_code UNIQUE (tenant_id, business_code, report_code) +); + +COMMENT ON TABLE report_report_config IS '报表配置表'; +COMMENT ON COLUMN report_report_config.tenant_id IS '租户ID'; +COMMENT ON COLUMN report_report_config.business_code IS '所属业务编码'; +COMMENT ON COLUMN report_report_config.report_code IS '报表编码(唯一标识)'; +COMMENT ON COLUMN report_report_config.report_name IS '报表名称'; +COMMENT ON COLUMN report_report_config.stat_table_name IS '统计宽表表名'; +COMMENT ON COLUMN report_report_config.stat_table_comment IS '统计宽表注释'; +COMMENT ON COLUMN report_report_config.date_field IS '日期字段名'; +COMMENT ON COLUMN report_report_config.primary_keys IS '主键字段列表(JSON数组)'; +COMMENT ON COLUMN report_report_config.conflict_keys IS '冲突键(用于ON CONFLICT upsert)'; +COMMENT ON COLUMN report_report_config.config IS '报表级配置(JSON)'; + +CREATE INDEX IF NOT EXISTS idx_report_business_code ON report_report_config (tenant_id, business_code); + +-- ---------------------------------------------------------- +-- 3. 报表字段配置表 (report_field_config) +-- 维度、指标、筛选字段 100% 配置化 +-- ---------------------------------------------------------- +CREATE TABLE IF NOT EXISTS report_field_config ( + id BIGSERIAL PRIMARY KEY, + tenant_id BIGINT NOT NULL DEFAULT 0, + business_code VARCHAR(64) NOT NULL, + report_code VARCHAR(64) NOT NULL, + field_code VARCHAR(64) NOT NULL, + field_name VARCHAR(128) NOT NULL, + field_type VARCHAR(32) NOT NULL, + data_type VARCHAR(32) NOT NULL DEFAULT 'STRING', + field_role VARCHAR(32) NOT NULL, + is_aggregatable BOOLEAN DEFAULT FALSE, + is_filterable BOOLEAN DEFAULT TRUE, + is_queryable BOOLEAN DEFAULT TRUE, + is_sortable BOOLEAN DEFAULT TRUE, + default_aggregate VARCHAR(32) DEFAULT '', + valid_aggregates JSONB DEFAULT '[]'::jsonb, + filter_operators JSONB DEFAULT '["=","!=",">","<",">=","<=","IN","LIKE","BETWEEN"]'::jsonb, + expression VARCHAR(512) DEFAULT '', + expression_type VARCHAR(32) DEFAULT '', + format_pattern VARCHAR(64) DEFAULT '', + unit VARCHAR(32) DEFAULT '', + dict_code VARCHAR(64) DEFAULT '', + sort_order INT DEFAULT 0, + group_name VARCHAR(64) DEFAULT '', + status VARCHAR(16) NOT NULL DEFAULT 'ACTIVE', + creator VARCHAR(64) DEFAULT '', + created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(), + updater VARCHAR(64) DEFAULT '', + updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(), + deleted_at TIMESTAMP WITH TIME ZONE, + + CONSTRAINT uk_business_report_field_code UNIQUE (tenant_id, business_code, report_code, field_code) +); + +COMMENT ON TABLE report_field_config IS '报表字段配置表(维度/指标/筛选)'; +COMMENT ON COLUMN report_field_config.tenant_id IS '租户ID'; +COMMENT ON COLUMN report_field_config.business_code IS '所属业务编码'; +COMMENT ON COLUMN report_field_config.report_code IS '所属报表编码'; +COMMENT ON COLUMN report_field_config.field_code IS '字段编码(唯一)'; +COMMENT ON COLUMN report_field_config.field_name IS '字段显示名称'; +COMMENT ON COLUMN report_field_config.field_type IS '字段类型: STRING/INT/FLOAT/DATE/DATETIME/JSONB'; +COMMENT ON COLUMN report_field_config.data_type IS '数据类型: DIMENSION维度/INDICATOR指标/FILTER筛选'; +COMMENT ON COLUMN report_field_config.field_role IS '字段角色: DIMENSION维度/INDICATOR指标/FILTER筛选/FILTER_ONLY仅筛选'; +COMMENT ON COLUMN report_field_config.is_aggregatable IS '是否可统计(sum/count/avg)'; +COMMENT ON COLUMN report_field_config.is_filterable IS '是否可筛选'; +COMMENT ON COLUMN report_field_config.is_queryable IS '是否可查询'; +COMMENT ON COLUMN report_field_config.is_sortable IS '是否可排序'; +COMMENT ON COLUMN report_field_config.default_aggregate IS '默认聚合方式: SUM/COUNT/AVG/MAX/MIN'; +COMMENT ON COLUMN report_field_config.valid_aggregates IS '支持的聚合方式(JSON数组)'; +COMMENT ON COLUMN report_field_config.filter_operators IS '支持的筛选操作符(JSON数组)'; +COMMENT ON COLUMN report_field_config.expression IS '计算表达式(如退款率=退款金额/销售额*100)'; +COMMENT ON COLUMN report_field_config.expression_type IS '表达式类型: CALCULATED衍生/ORIGINAL原始'; +COMMENT ON COLUMN report_field_config.format_pattern IS '格式化模式(如 #,##0.00%)'; +COMMENT ON COLUMN report_field_config.unit IS '单位(如元/件/%)'; +COMMENT ON COLUMN report_field_config.dict_code IS '字典编码(用于下拉选项)'; +COMMENT ON COLUMN report_field_config.sort_order IS '排序顺序'; +COMMENT ON COLUMN report_field_config.group_name IS '字段分组名称'; + +CREATE INDEX IF NOT EXISTS idx_field_business_report ON report_field_config (business_code, report_code); +CREATE INDEX IF NOT EXISTS idx_field_data_type ON report_field_config (data_type); +CREATE INDEX IF NOT EXISTS idx_field_field_role ON report_field_config (field_role); + +-- ---------------------------------------------------------- +-- 4. 数据抽取配置表 (extract_config) +-- 100% 配置化，不硬编码任何抽取逻辑 +-- ---------------------------------------------------------- +CREATE TABLE IF NOT EXISTS report_extract_config ( + id BIGSERIAL PRIMARY KEY, + tenant_id BIGINT NOT NULL DEFAULT 0, + business_code VARCHAR(64) NOT NULL, + report_code VARCHAR(64) NOT NULL, + extract_code VARCHAR(64) NOT NULL, + extract_name VARCHAR(128) NOT NULL, + source_table_name VARCHAR(128) NOT NULL, + source_table_alias VARCHAR(64) DEFAULT '', + target_table_name VARCHAR(128) NOT NULL, + is_enabled BOOLEAN DEFAULT TRUE, + extract_type VARCHAR(32) NOT NULL DEFAULT 'FULL', + extract_mode VARCHAR(32) NOT NULL DEFAULT 'DIRECT', + extract_key_field VARCHAR(64) DEFAULT '', + extract_key_format VARCHAR(64) DEFAULT '', + group_by_fields JSONB DEFAULT '[]'::jsonb, + filter_expression TEXT DEFAULT '', + join_configs JSONB DEFAULT '[]'::jsonb, + field_mappings JSONB DEFAULT '[]'::jsonb, + transform_rules JSONB DEFAULT '[]'::jsonb, + batch_size INT DEFAULT 1000, + status VARCHAR(16) NOT NULL DEFAULT 'ACTIVE', + creator VARCHAR(64) DEFAULT '', + created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(), + updater VARCHAR(64) DEFAULT '', + updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(), + deleted_at TIMESTAMP WITH TIME ZONE, + + CONSTRAINT uk_business_report_extract_code UNIQUE (tenant_id, business_code, report_code, extract_code) +); + +COMMENT ON TABLE report_extract_config IS '数据抽取配置表'; +COMMENT ON COLUMN report_extract_config.tenant_id IS '租户ID'; +COMMENT ON COLUMN report_extract_config.business_code IS '所属业务编码'; +COMMENT ON COLUMN report_extract_config.report_code IS '所属报表编码'; +COMMENT ON COLUMN report_extract_config.extract_code IS '抽取配置编码'; +COMMENT ON COLUMN report_extract_config.extract_name IS '抽取配置名称'; +COMMENT ON COLUMN report_extract_config.source_table_name IS '源表表名'; +COMMENT ON COLUMN report_extract_config.source_table_alias IS '源表别名'; +COMMENT ON COLUMN report_extract_config.target_table_name IS '目标表表名(统计宽表)'; +COMMENT ON COLUMN report_extract_config.extract_type IS '抽取类型: FULL全量/INCREMENTAL增量'; +COMMENT ON COLUMN report_extract_config.extract_mode IS '抽取模式: DIRECT逐行抽取/AGGREGATE按GROUP BY聚合'; +COMMENT ON COLUMN report_extract_config.extract_key_field IS '增量抽取关键字段(如updated_at)'; +COMMENT ON COLUMN report_extract_config.extract_key_format IS '关键字段格式(如yyyy-MM-dd HH:mm:ss)'; +COMMENT ON COLUMN report_extract_config.group_by_fields IS 'AGGREGATE模式下的GROUP BY字段(JSON数组)'; +COMMENT ON COLUMN report_extract_config.filter_expression IS '抽取条件表达式'; +COMMENT ON COLUMN report_extract_config.join_configs IS '关联配置(JSON数组)'; +COMMENT ON COLUMN report_extract_config.field_mappings IS '字段映射配置(JSON数组)'; +COMMENT ON COLUMN report_extract_config.transform_rules IS '转换规则(JSON数组)'; +COMMENT ON COLUMN report_extract_config.batch_size IS '批次大小'; + +CREATE INDEX IF NOT EXISTS idx_extract_business_report ON report_extract_config (business_code, report_code); + +-- ---------------------------------------------------------- +-- 5. 抽取记录表 (extract_log) - 用于幂等和追踪 +-- ---------------------------------------------------------- +CREATE TABLE IF NOT EXISTS report_extract_log ( + id BIGSERIAL PRIMARY KEY, + tenant_id BIGINT NOT NULL DEFAULT 0, + business_code VARCHAR(64) NOT NULL, + report_code VARCHAR(64) NOT NULL, + extract_code VARCHAR(64) NOT NULL, + stat_date VARCHAR(16) NOT NULL, + extract_type VARCHAR(32) NOT NULL, + status VARCHAR(16) NOT NULL DEFAULT 'RUNNING', + total_count INT DEFAULT 0, + success_count INT DEFAULT 0, + fail_count INT DEFAULT 0, + start_time TIMESTAMP WITH TIME ZONE DEFAULT NOW(), + end_time TIMESTAMP WITH TIME ZONE, + error_message TEXT DEFAULT '', + executor VARCHAR(64) DEFAULT '', + created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(), + updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(), + + CONSTRAINT uk_extract_keys UNIQUE (tenant_id, business_code, report_code, extract_code, stat_date) +); + +COMMENT ON TABLE report_extract_log IS '抽取记录表(幂等追踪)'; +COMMENT ON COLUMN report_extract_log.tenant_id IS '租户ID'; +COMMENT ON COLUMN report_extract_log.business_code IS '业务编码'; +COMMENT ON COLUMN report_extract_log.report_code IS '报表编码'; +COMMENT ON COLUMN report_extract_log.extract_code IS '抽取配置编码'; +COMMENT ON COLUMN report_extract_log.stat_date IS '统计日期(yyyy-MM-dd)'; +COMMENT ON COLUMN report_extract_log.extract_type IS '抽取类型: FULL/INCREMENTAL'; +COMMENT ON COLUMN report_extract_log.status IS '状态: RUNNING运行中/SUCCESS成功/FAILED失败'; +COMMENT ON COLUMN report_extract_log.total_count IS '总记录数'; +COMMENT ON COLUMN report_extract_log.success_count IS '成功记录数'; +COMMENT ON COLUMN report_extract_log.fail_count IS '失败记录数'; +CREATE INDEX IF NOT EXISTS idx_extract_log_business_report ON report_extract_log (business_code, report_code); +CREATE INDEX IF NOT EXISTS idx_extract_log_stat_date ON report_extract_log (stat_date); + +-- ============================================================ +-- 字段映射配置结构说明 (field_mappings JSONB) +-- ============================================================ +-- [ +-- { +-- "source_field": "order_id", +-- "target_field": "order_id", +-- "field_type": "STRING", +-- "default_value": "", +-- "transform_rule": { +-- "type": "DIRECT/MAPPING/CALCULATE", +-- "expression": "" +-- } +-- }, +-- { +-- "source_field": "sale_amount", +-- "target_field": "sale_amount_yuan", +-- "field_type": "FLOAT", +-- "default_value": 0, +-- "transform_rule": { +-- "type": "CALCULATE", +-- "expression": "source_value / 100" +-- } +-- } +-- ] + +-- ============================================================ +-- 转换规则配置结构说明 (transform_rules JSONB) +-- ============================================================ +-- [ +-- { +-- "rule_code": "status_mapping", +-- "rule_type": "MAPPING", +-- "source_field": "order_status", +-- "target_field": "status_name", +-- "mapping": { +-- "PENDING": "待支付", +-- "PAID": "已支付", +-- "SHIPPED": "已发货", +-- "COMPLETED": "已完成", +-- "REFUNDED": "已退款" +-- } +-- }, +-- { +-- "rule_code": "date_format", +-- "rule_type": "FORMAT", +-- "source_field": "create_time", +-- "target_field": "stat_date", +-- "format": "yyyy-MM-dd" +-- } +-- ] + +-- ============================================================ +-- 关联配置结构说明 (join_configs JSONB) +-- ============================================================ +-- [ +-- { +-- "join_table": "product_info", +-- "join_alias": "p", +-- "join_type": "LEFT", +-- "join_condition": "s.product_id = p.id", +-- "field_mappings": [ +-- {"source_field": "p.category_name", "target_field": "category_name"} +-- ] +-- } +-- ] diff --git a/sql/seed_data_dingtalk.sql b/sql/seed_data_dingtalk.sql new file mode 100644 index 0000000..0d78c2e --- /dev/null +++ b/sql/seed_data_dingtalk.sql @@ -0,0 +1,229 @@ +-- ============================================= +-- 钉钉开放平台初始化数据 +-- +-- 执行后需配置 token（OAUTH2 读取 token 列，非 api_key）： +-- UPDATE api_datasource_platform SET token = '你的oapi access_token' +-- WHERE platform_code = 'dingtalk'; +-- ============================================= + +-- 1. 创建钉钉平台 +INSERT INTO api_datasource_platform ( + tenant_id, creator, created_at, updater, updated_at, + platform_code, platform_name, description, status, + api_base_url, auth_type, + auth_config, + rate_limit_per_minute, rate_limit_per_hour, + concurrency_limit, request_timeout_ms, max_retries, retry_delay_ms +) VALUES ( + 1, 'admin', NOW(), 'admin', NOW(), + 'dingtalk', '钉钉', '钉钉开放平台数据同步', 'ACTIVE', + 'https://oapi.dingtalk.com', 'OAUTH2', + '{ + "token_in_query": true, + "query_key": "access_token" + }'::jsonb, + 60, 3600, 5, 30000, 3, 1000 +); + +-- 2. 部门列表（递归遍历全量部门树） +-- 先从根（不传 dept_id）获取一级部门，再对每个子部门递归调用获取下级 +-- 请求：POST，access_token 在 URL 查询参数中 +-- 响应：{"errcode":0,"errmsg":"ok","result":[{...}]} +INSERT INTO api_interface ( + tenant_id, creator, created_at, updater, updated_at, + platform_id, name, code, url, method, status, auth_type, + request_config, response_config, table_definition +) VALUES ( + 1, 'admin', NOW(), 'admin', NOW(), + (SELECT id FROM api_datasource_platform WHERE platform_code = 'dingtalk'), + '部门列表', 'department_list', + '/topapi/v2/department/listsub', 'POST', 'active', 'inherit', + '{ + "parameters_location": "query", + "page_param": "cursor", + "page_size_param": "pageSize", + "language": "zh_CN", + "recursive": { + "key_field": "dept_id", + "target_param": "dept_id" + }, + "max_recursive_depth": 20 + }'::jsonb, + '{ + "success_field": "errcode", + "success_value": 0, + "message_field": "errmsg", + "list_path": "result" + }'::jsonb, + '{ + "table_name": "dingtalk_department", + "columns": [ + {"name": "dept_id", "type": "BIGINT", "comment": "部门ID"}, + {"name": "name", "type": "VARCHAR(300)", "comment": "部门名称"}, + {"name": "parent_id", "type": "BIGINT", "comment": "父部门ID"}, + {"name": "create_dept_group", "type": "BOOLEAN", "comment": "是否同步创建关联企业群"}, + {"name": "auto_add_user", "type": "BOOLEAN", "comment": "新人是否自动加入部门群"} + ], + "conflict_keys": ["dept_id"] + }'::jsonb +); + +-- 3. 部门用户列表（prefetch→department_list，游标分页） +-- 先递归获取全量部门树，再对每个部门分页拉取用户 +-- dept_id 为必填，通过 prefetch 从部门列表注入 +-- 响应：result.list（用户数组），result.has_more（是否还有更多），result.next_cursor（下一游标） +INSERT INTO api_interface ( + tenant_id, creator, created_at, updater, updated_at, + platform_id, name, code, url, method, status, auth_type, + request_config, response_config, table_definition +) VALUES ( + 1, 'admin', NOW(), 'admin', NOW(), + (SELECT id FROM api_datasource_platform WHERE platform_code = 'dingtalk'), + '部门用户列表', 'user_list', + '/topapi/v2/user/list', 'POST', 'active', 'inherit', + '{ + "parameters_location": "query", + "page_param": "cursor", + "page_size_param": "size", + "cursor_pagination": true, + "size": 100, + "order_field": "modify_desc", + "language": "zh_CN", + "initial_cursor": 0, + "prefetch": { + "url": "/topapi/v2/department/listsub", + "method": "POST", + "response_path": "result", + "target_param": "dept_id", + "value_field": "dept_id" + } + }'::jsonb, + '{ + "success_field": "errcode", + "success_value": 0, + "message_field": "errmsg", + "list_path": "result.list", + "cursor_field": "result.next_cursor", + "has_more_field": "result.has_more" + }'::jsonb, + '{ + "table_name": "dingtalk_user", + "columns": [ + {"name": "userid", "type": "VARCHAR(100)", "comment": "用户userId"}, + {"name": "unionid", "type": "VARCHAR(200)", "comment": "用户唯一标识"}, + {"name": "name", "type": "VARCHAR(200)", "comment": "用户姓名"}, + {"name": "avatar", "type": "TEXT", "comment": "头像地址"}, + {"name": "state_code", "type": "VARCHAR(20)", "comment": "国际电话区号"}, + {"name": "mobile", "type": "VARCHAR(50)", "comment": "手机号码"}, + {"name": "hide_mobile", "type": "BOOLEAN", "comment": "是否号码隐藏"}, + {"name": "telephone", "type": "VARCHAR(50)", "comment": "分机号"}, + {"name": "job_number", "type": "VARCHAR(100)", "comment": "员工工号"}, + {"name": "title", "type": "VARCHAR(200)", "comment": "职位"}, + {"name": "email", "type": "VARCHAR(200)", "comment": "员工邮箱"}, + {"name": "org_email", "type": "VARCHAR(200)", "comment": "企业邮箱"}, + {"name": "work_place", "type": "VARCHAR(300)", "comment": "办公地点"}, + {"name": "remark", "type": "TEXT", "comment": "备注"}, + {"name": "dept_id_list", "type": "JSONB", "comment": "所属部门id列表"}, + {"name": "dept_order", "type": "INT", "comment": "员工在部门中的排序"}, + {"name": "extension", "type": "TEXT", "comment": "扩展属性"}, + {"name": "hired_date", "type": "BIGINT", "comment": "入职时间"}, + {"name": "active", "type": "BOOLEAN", "comment": "是否激活钉钉"}, + {"name": "admin", "type": "BOOLEAN", "comment": "是否企业管理员"}, + {"name": "boss", "type": "BOOLEAN", "comment": "是否企业老板"}, + {"name": "leader", "type": "BOOLEAN", "comment": "是否部门主管"}, + {"name": "exclusive_account", "type": "BOOLEAN", "comment": "是否企业账号"} + ], + "conflict_keys": ["userid"] + }'::jsonb +); + +-- 4. 考勤数据查询（单个用户单日，prefetch→user_list） +-- 需先同步 user_list 获取全量用户，再对每个用户调用此接口查询考勤数据 +-- userid 通过 prefetch 从 user_list 注入，work_date 固定为当天日期 +-- 响应：result 对象（包含 attendance_result_list/check_record_list/approve_list 等） +-- 注意：当前代码支持两级 prefetch 链（即本接口 prefetch→user_list， +-- user_list 自有 prefetch→department_list），但 user_list 的 dept_id 依赖 +-- 在作为 prefetch 来源时不会自动解析。若需全量同步所有用户的考勤数据， +-- 建议先手动同步 user_list 和 department_list，再同步此接口。 +-- 使用前请将 work_date 更新为目标日期，如 '2026-06-03' +INSERT INTO api_interface ( + tenant_id, creator, created_at, updater, updated_at, + platform_id, name, code, url, method, status, auth_type, + request_config, response_config, table_definition +) VALUES ( + 1, 'admin', NOW(), 'admin', NOW(), + (SELECT id FROM api_datasource_platform WHERE platform_code = 'dingtalk'), + '考勤数据查询', 'attendance_getupdatedata', + '/topapi/attendance/getupdatedata', 'POST', 'active', 'inherit', + '{ + "parameters_location": "query", + "prefetch": { + "url": "/topapi/v2/user/list", + "method": "POST", + "response_path": "result.list", + "target_param": "userid", + "value_field": "userid" + }, + "work_date": "2026-06-03" + }'::jsonb, + '{ + "success_field": "errcode", + "success_value": 0, + "message_field": "errmsg", + "list_path": "result", + "single_record": true + }'::jsonb, + '{ + "table_name": "dingtalk_attendance", + "columns": [ + {"name": "userid", "type": "VARCHAR(100)", "comment": "用户userId"}, + {"name": "work_date", "type": "VARCHAR(30)", "comment": "查询日期"}, + {"name": "corpId", "type": "VARCHAR(100)", "comment": "企业corpId"}, + {"name": "attendance_result_list", "type": "JSONB", "comment": "打卡结果列表"}, + {"name": "check_record_list", "type": "JSONB", "comment": "打卡详情列表"}, + {"name": "approve_list", "type": "JSONB", "comment": "审批单列表"}, + {"name": "rest_time_vo_list", "type": "JSONB", "comment": "班次内休息信息"}, + {"name": "rest_end_time", "type": "BIGINT", "comment": "休息结束时间"}, + {"name": "rest_begin_time", "type": "BIGINT", "comment": "休息开始时间"} + ], + "conflict_keys": ["userid", "work_date"] + }'::jsonb +); + +-- 5. 角色列表（偏移量分页，offset/size + hasMore） +-- 响应：result.list（角色组数组），result.hasMore（是否还有更多） +-- 每个角色组内包含 roles 数组（{id, name}）你 +INSERT INTO api_interface ( + tenant_id, creator, created_at, updater, updated_at, + platform_id, name, code, url, method, status, auth_type, + request_config, response_config, table_definition +) VALUES ( + 1, 'admin', NOW(), 'admin', NOW(), + (SELECT id FROM api_datasource_platform WHERE platform_code = 'dingtalk'), + '角色列表', 'role_list', + '/topapi/role/list', 'POST', 'active', 'inherit', + '{ + "parameters_location": "query", + "page_param": "offset", + "page_size_param": "size", + "size": 200, + "offset": 0, + "pagination_mode": "offset" + }'::jsonb, + '{ + "success_field": "errcode", + "success_value": 0, + "message_field": "errmsg", + "list_path": "result.list", + "has_more_field": "result.has_more" + }'::jsonb, + '{ + "table_name": "dingtalk_role", + "columns": [ + {"name": "groupId", "type": "BIGINT", "comment": "角色组ID"}, + {"name": "name", "type": "VARCHAR(200)", "comment": "角色组名称"}, + {"name": "roles", "type": "JSONB", "comment": "角色列表"} + ], + "conflict_keys": ["groupId"] + }'::jsonb +); diff --git a/sql/seed_data_dingtalk_hrm.sql b/sql/seed_data_dingtalk_hrm.sql new file mode 100644 index 0000000..d047d63 --- /dev/null +++ b/sql/seed_data_dingtalk_hrm.sql @@ -0,0 +1,69 @@ +-- ============================================= +-- 钉钉智能人事平台初始化数据 +-- 注意：此平台使用 x-acs-dingtalk-access-token Header 认证 +-- API 基础地址为 https://api.dingtalk.com（与 oapi 不同） +-- +-- 执行后需配置 token（OAUTH2 读取 token 列，非 api_key）： +-- UPDATE api_datasource_platform SET token = '你的hrm access_token' +-- WHERE platform_code = 'dingtalk_hrm'; +-- ============================================= + +-- 1. 创建钉钉智能人事平台 +INSERT INTO api_datasource_platform ( + tenant_id, creator, created_at, updater, updated_at, + platform_code, platform_name, description, status, + api_base_url, auth_type, + auth_config, + rate_limit_per_minute, rate_limit_per_hour, + concurrency_limit, request_timeout_ms, max_retries, retry_delay_ms +) VALUES ( + 1, 'admin', NOW(), 'admin', NOW(), + 'dingtalk_hrm', '钉钉智能人事', '钉钉智能人事数据同步（header 认证）', 'ACTIVE', + 'https://api.dingtalk.com', 'OAUTH2', + '{ + "token_in_query": false, + "header_name": "x-acs-dingtalk-access-token", + "header_format": "{token}" + }'::jsonb, + 60, 3600, 5, 30000, 3, 1000 +); + +-- 2. 企业职位列表（游标分页，hasMore 判结束） +-- 响应：list（职位数组），nextToken（游标），hasMore（是否还有更多） +-- 注意：此接口成功时无 errcode 字段，parseRespExt 在找不到 success_field 时自动跳过检查 +INSERT INTO api_interface ( + tenant_id, creator, created_at, updater, updated_at, + platform_id, name, code, url, method, status, auth_type, + request_config, response_config, table_definition +) VALUES ( + 1, 'admin', NOW(), 'admin', NOW(), + (SELECT id FROM api_datasource_platform WHERE platform_code = 'dingtalk_hrm'), + '企业职位列表', 'position_list', + '/v1.0/hrm/positions/query', 'POST', 'active', 'inherit', + '{ + "cursor_pagination": true, + "page_param": "nextToken", + "page_size_param": "maxResults", + "pageSize": 200, + "initial_cursor": 0, + "parameters_location": "query" + }'::jsonb, + '{ + "list_path": "list", + "cursor_field": "nextToken", + "has_more_field": "hasMore" + }'::jsonb, + '{ + "table_name": "dingtalk_position", + "columns": [ + {"name": "positionId", "type": "VARCHAR(100)", "comment": "职位ID"}, + {"name": "positionName", "type": "VARCHAR(300)", "comment": "职位名称"}, + {"name": "positionCategoryId", "type": "VARCHAR(100)", "comment": "职位类别ID"}, + {"name": "jobId", "type": "VARCHAR(100)", "comment": "所属职务ID"}, + {"name": "positionDes", "type": "TEXT", "comment": "职位描述"}, + {"name": "rankIdList", "type": "JSONB", "comment": "职级ID列表"}, + {"name": "status", "type": "INT", "comment": "职位状态 0启用 1停用"} + ], + "conflict_keys": ["positionId"] + }'::jsonb +); diff --git a/sql/seed_data_dingtalk_salary.sql b/sql/seed_data_dingtalk_salary.sql new file mode 100644 index 0000000..55b0dc2 --- /dev/null +++ b/sql/seed_data_dingtalk_salary.sql @@ -0,0 +1,175 @@ +-- ============================================= +-- 钉钉智能薪酬平台初始化数据 +-- +-- 认证方式：app-id + signature 头部 +-- 签名算法：MD5(request_body_string + app_secret).toUpperCase() +-- +-- 执行后需在 auth_config 中配置 app_id 和 app_secret： +-- UPDATE api_datasource_platform SET auth_config = jsonb_set( +-- jsonb_set(auth_config, '{app_id}', '"你的app-id"'), +-- '{app_secret}', '"你的app-secret"' +-- ) WHERE platform_code = 'dingtalk_salary'; +-- ============================================= + +-- 1. 创建钉钉智能薪酬平台 +INSERT INTO api_datasource_platform ( + tenant_id, creator, created_at, updater, updated_at, + platform_code, platform_name, description, status, + api_base_url, auth_type, + auth_config, + rate_limit_per_minute, rate_limit_per_hour, + concurrency_limit, request_timeout_ms, max_retries, retry_delay_ms +) VALUES ( + 1, 'admin', NOW(), 'admin', NOW(), + 'dingtalk_salary', '钉钉智能薪酬', '钉钉智能薪酬数据同步（app-id + signature 头部认证）', 'ACTIVE', + 'https://salary.eapps.dingtalkcloud.com', 'APP_SIGNATURE', + '{ + "app_id": "", + "app_secret": "", + "sign_algorithm": "md5_upper_body", + "app_id_header": "app-id", + "sign_header": "signature" + }'::jsonb, + 60, 3600, 3, 30000, 3, 1000 +); + +-- 2. 部门列表（无分页，单次请求返回全部部门） +-- 无请求入参，直接 POST 即可 +-- 响应：{"code":"200","success":true,"data":[{"name":"研发部","value":"93639012","ext":"-1"},...]} +INSERT INTO api_interface ( + tenant_id, creator, created_at, updater, updated_at, + platform_id, name, code, url, method, status, auth_type, + request_config, response_config, table_definition +) VALUES ( + 1, 'admin', NOW(), 'admin', NOW(), + (SELECT id FROM api_datasource_platform WHERE platform_code = 'dingtalk_salary'), + '公司部门列表', 'salary_dept_list', + '/oapi/salary/dept/list', 'POST', 'active', 'inherit', + '{}'::jsonb, + '{ + "success_field": "code", + "success_value": 200, + "message_field": "msg", + "list_path": "data" + }'::jsonb, + '{ + "table_name": "dingtalk_salary_dept", + "columns": [ + {"name": "name", "type": "VARCHAR(300)", "comment": "部门名称"}, + {"name": "value", "type": "VARCHAR(100)", "comment": "部门ID"}, + {"name": "ext", "type": "VARCHAR(100)", "comment": "父部门ID，-1为根部门"} + ], + "conflict_keys": ["value"] + }'::jsonb +); + +-- 3. 按部门导出当月薪资报表（prefetch→salary_dept_list） +-- 先通过 dept/list 获取全量部门列表，再对每个部门调用此接口 +-- deptId 通过 prefetch 注入，statisticsMonth 通过 row_inject 注入到每行 +-- 使用前请将 statisticsMonth 更新为目标月份，如 "2026-06" +INSERT INTO api_interface ( + tenant_id, creator, created_at, updater, updated_at, + platform_id, name, code, url, method, status, auth_type, + request_config, response_config, table_definition +) VALUES ( + 1, 'admin', NOW(), 'admin', NOW(), + (SELECT id FROM api_datasource_platform WHERE platform_code = 'dingtalk_salary'), + '按部门薪资报表', 'salary_statistics', + '/oapi/salary/statistics/dept', 'POST', 'active', 'inherit', + '{ + "statisticsMonth": "2026-06", + "row_inject": ["statisticsMonth"], + "prefetch": { + "url": "/oapi/salary/dept/list", + "method": "POST", + "response_path": "data", + "target_param": "deptId", + "value_field": "value" + } + }'::jsonb, + '{ + "success_field": "code", + "success_value": 200, + "message_field": "msg", + "list_path": "data" + }'::jsonb, + '{ + "table_name": "dingtalk_salary_statistics", + "columns": [ + {"name": "deptId", "type": "BIGINT", "comment": "部门ID"}, + {"name": "statisticsMonth", "type": "VARCHAR(7)", "comment": "薪资月(yyyy-MM)"}, + {"name": "name", "type": "VARCHAR(200)", "comment": "报表项名称"}, + {"name": "value", "type": "VARCHAR(100)", "comment": "报表项数据"} + ], + "conflict_keys": ["deptId", "statisticsMonth", "name"] + }'::jsonb +); + +-- 4. 人力成本报表（无分页，单次请求返回指定月份的人力成本汇总） +-- calBizId 为空时默认为当前月份，格式为 yyyyMM + "M"，如 "202606M" +-- 使用前可将 calBizId 更新为目标月份 +-- 响应：data.sumStatisticsData 包含各成本项（itemId/itemName/itemDesc/sValue/value） +INSERT INTO api_interface ( + tenant_id, creator, created_at, updater, updated_at, + platform_id, name, code, url, method, status, auth_type, + request_config, response_config, table_definition +) VALUES ( + 1, 'admin', NOW(), 'admin', NOW(), + (SELECT id FROM api_datasource_platform WHERE platform_code = 'dingtalk_salary'), + '人力成本报表', 'salary_statistic_report', + '/oapi/salary/statistic/report/data', 'POST', 'active', 'inherit', + '{ + "calBizId": "202606M" + }'::jsonb, + '{ + "success_field": "code", + "success_value": 200, + "message_field": "msg", + "list_path": "data", + "single_record": true + }'::jsonb, + '{ + "table_name": "dingtalk_salary_statistic_report", + "columns": [ + {"name": "calBizId", "type": "VARCHAR(20)", "comment": "薪资月份"}, + {"name": "sumStatisticsData", "type": "JSONB", "comment": "人力成本报表数据"} + ], + "conflict_keys": ["calBizId"] + }'::jsonb +); + +-- 5. 按薪资组获取人力成本报表（无分页，单次请求返回指定薪资组的人力成本汇总） +-- salaryGroupName 为必填，使用前需更新为实际的薪资组名称 +-- calBizId 为空时默认为当前月份，格式为 yyyyMM + "M" +-- row_inject 会将请求中的 salaryGroupName 注入到每行，便于区分不同薪资组的报表 +INSERT INTO api_interface ( + tenant_id, creator, created_at, updater, updated_at, + platform_id, name, code, url, method, status, auth_type, + request_config, response_config, table_definition +) VALUES ( + 1, 'admin', NOW(), 'admin', NOW(), + (SELECT id FROM api_datasource_platform WHERE platform_code = 'dingtalk_salary'), + '按薪资组人力成本报表', 'salary_statistic_report_group', + '/oapi/salary/statistic/report/groupData', 'POST', 'active', 'inherit', + '{ + "calBizId": "202606M", + "salaryGroupName": "", + "row_inject": ["salaryGroupName"] + }'::jsonb, + '{ + "success_field": "code", + "success_value": 200, + "message_field": "msg", + "list_path": "data", + "single_record": true + }'::jsonb, + '{ + "table_name": "dingtalk_salary_statistic_report_group", + "columns": [ + {"name": "calBizId", "type": "VARCHAR(20)", "comment": "薪资月份"}, + {"name": "salaryGroupName", "type": "VARCHAR(200)", "comment": "薪资组名称"}, + {"name": "sumStatisticsData", "type": "JSONB", "comment": "人力成本报表数据"}0 + ], + "conflict_keys": ["calBizId", "salaryGroupName"] + }'::jsonb +); diff --git a/sql/seed_data_report_kuaishou.sql b/sql/seed_data_report_kuaishou.sql new file mode 100644 index 0000000..9908fb1 --- /dev/null +++ b/sql/seed_data_report_kuaishou.sql @@ -0,0 +1,435 @@ +-- ============================================================ +-- 快手电商 - 报表系统初始化数据 +-- 基于 seed_data_kuaishou.sql 中实际表结构生成 +-- 覆盖报表: 订单 / 商品 / 售后 / 分销订单 / 代发订单 +-- ============================================================ + +-- ============================================= +-- 1. 业务配置 (report_business_config) +-- ============================================= +INSERT INTO report_business_config (tenant_id, business_code, business_name, description, status, config, creator, created_at, updater, updated_at) VALUES +(1, 'kuaishou_ecommerce', '快手电商', '快手电商平台订单、商品、售后、分销数据分析', 'ACTIVE', + '{"platform":"kuaishou","timezone":"Asia/Shanghai"}'::jsonb, + 'admin', NOW(), 'admin', NOW()) +ON CONFLICT (tenant_id, business_code) DO NOTHING; + + +-- ============================================= +-- 2. 报表配置 (report_report_config) +-- ============================================= + +-- 2.1 订单分析 +INSERT INTO report_report_config (tenant_id, business_code, report_code, report_name, description, status, stat_table_name, stat_table_comment, date_field, primary_keys, conflict_keys, config, creator, created_at, updater, updated_at) VALUES +(1, 'kuaishou_ecommerce', 'order_analysis', '订单分析', '基于 kuaishou_order_list 的订单维度聚合分析，支持按日期/状态/地区/支付方式/渠道等维度', 'ACTIVE', + 'kuaishou_order_stat', '快手订单统计宽表（聚合自 kuaishou_order_list）', + 'stat_date', + '["id"]'::jsonb, + '["stat_date","oid"]'::jsonb, + '{"defaultTimeRange":"last_30_days","defaultDimensions":["stat_date"],"defaultIndicators":[{"field":"order_count","aggregate":"COUNT"},{"field":"total_fee","aggregate":"SUM"}]}'::jsonb, + 'admin', NOW(), 'admin', NOW()) +ON CONFLICT (tenant_id, business_code, report_code) DO NOTHING; + +-- 2.2 商品分析 +INSERT INTO report_report_config (tenant_id, business_code, report_code, report_name, description, status, stat_table_name, stat_table_comment, date_field, primary_keys, conflict_keys, config, creator, created_at, updater, updated_at) VALUES +(1, 'kuaishou_ecommerce', 'item_analysis', '商品分析', '基于 kuaishou_item_list 的商品维度销量、销售额分析，支持按类目/类型/上下架状态等维度', 'ACTIVE', + 'kuaishou_item_stat', '快手商品统计宽表（聚合自 kuaishou_item_list）', + 'stat_date', + '["id"]'::jsonb, + '["stat_date","kwai_item_id","category_id"]'::jsonb, + '{"defaultTimeRange":"last_30_days","defaultDimensions":["stat_date","category_name"],"defaultIndicators":[{"field":"sales_volume","aggregate":"SUM"},{"field":"sales_amount","aggregate":"SUM"}]}'::jsonb, + 'admin', NOW(), 'admin', NOW()) +ON CONFLICT (tenant_id, business_code, report_code) DO NOTHING; + +-- 2.3 售后分析 +INSERT INTO report_report_config (tenant_id, business_code, report_code, report_name, description, status, stat_table_name, stat_table_comment, date_field, primary_keys, conflict_keys, config, creator, created_at, updater, updated_at) VALUES +(1, 'kuaishou_ecommerce', 'refund_analysis', '售后分析', '基于 kuaishou_refund_list 的售后维度分析，退款金额/退款率等核心指标', 'ACTIVE', + 'kuaishou_refund_stat', '快手售后统计宽表（聚合自 kuaishou_refund_list）', + 'stat_date', + '["id"]'::jsonb, + '["stat_date","refund_id"]'::jsonb, + '{"defaultTimeRange":"last_30_days","defaultDimensions":["stat_date","refund_type"],"defaultIndicators":[{"field":"refund_count","aggregate":"COUNT"},{"field":"refund_fee","aggregate":"SUM"}]}'::jsonb, + 'admin', NOW(), 'admin', NOW()) +ON CONFLICT (tenant_id, business_code, report_code) DO NOTHING; + +-- 2.4 分销订单分析 +INSERT INTO report_report_config (tenant_id, business_code, report_code, report_name, description, status, stat_table_name, stat_table_comment, date_field, primary_keys, conflict_keys, config, creator, created_at, updater, updated_at) VALUES +(1, 'kuaishou_ecommerce', 'cps_order_analysis', '分销订单分析', '基于 kuaishou_cps_order_list 的分销订单维度分析，含分销收入/佣金等指标', 'ACTIVE', + 'kuaishou_cps_order_stat', '快手分销订单统计宽表（聚合自 kuaishou_cps_order_list）', + 'stat_date', + '["id"]'::jsonb, + '["stat_date","oid"]'::jsonb, + '{"defaultTimeRange":"last_30_days","defaultDimensions":["stat_date","status"],"defaultIndicators":[{"field":"order_count","aggregate":"COUNT"},{"field":"estimated_income","aggregate":"SUM"}]}'::jsonb, + 'admin', NOW(), 'admin', NOW()) +ON CONFLICT (tenant_id, business_code, report_code) DO NOTHING; + +-- 2.5 代发订单分析 +INSERT INTO report_report_config (tenant_id, business_code, report_code, report_name, description, status, stat_table_name, stat_table_comment, date_field, primary_keys, conflict_keys, config, creator, created_at, updater, updated_at) VALUES +(1, 'kuaishou_ecommerce', 'dropshipping_analysis', '代发订单分析', '基于 kuaishou_dropshipping_order_list 的代发订单维度分析，按厂家/物流/地区等维度', 'ACTIVE', + 'kuaishou_dropshipping_stat', '快手代发订单统计宽表（聚合自 kuaishou_dropshipping_order_list）', + 'stat_date', + '["id"]'::jsonb, + '["stat_date","dropshipping_order_code"]'::jsonb, + '{"defaultTimeRange":"last_30_days","defaultDimensions":["stat_date","dropshipping_status"],"defaultIndicators":[{"field":"order_count","aggregate":"COUNT"}]}'::jsonb, + 'admin', NOW(), 'admin', NOW()) +ON CONFLICT (tenant_id, business_code, report_code) DO NOTHING; + + +-- ============================================= +-- 3. 字段配置 (report_field_config) +-- ============================================= + +-- ---------------------------------------------------------- +-- 3.1 订单分析 - 字段配置 +-- 源表: kuaishou_order_list +-- 实际列: oid, status, createTime, updateTime, payTime, sendTime, recvTime, +-- refundTime, totalFee, expressFee, discountFee, originalPrice, +-- buyerOpenId, sellerOpenId, buyerNick, sellerNick, remark, itemTitle, +-- num, price, activityType, cpsType, payType, payChannel, channel, +-- commentStatus, priorityDelivery, carrierType, carrierId, +-- province, city, district, provinceCode, cityCode, districtCode +-- ---------------------------------------------------------- + +-- 维度字段 +INSERT INTO report_field_config (tenant_id, business_code, report_code, field_code, field_name, field_type, data_type, field_role, is_aggregatable, is_filterable, is_queryable, is_sortable, default_aggregate, valid_aggregates, filter_operators, expression, expression_type, format_pattern, unit, dict_code, sort_order, group_name, status, creator, created_at, updater, updated_at) VALUES +(1, 'kuaishou_ecommerce', 'order_analysis', 'stat_date', '统计日期', 'DATE', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=",">=","<=","BETWEEN"]'::jsonb, '', '', 'yyyy-MM-dd', '', '', 1, '时间维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'order_analysis', 'oid', '订单ID', 'BIGINT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', '', 2, '订单维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'order_analysis', 'status', '订单状态', 'INT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', 'order_status_dict', 3, '订单维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'order_analysis', 'province', '省份', 'VARCHAR', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN","LIKE"]'::jsonb, '', '', '', '', '', 4, '地域维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'order_analysis', 'city', '城市', 'VARCHAR', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN","LIKE"]'::jsonb, '', '', '', '', '', 5, '地域维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'order_analysis', 'district', '区县', 'VARCHAR', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN","LIKE"]'::jsonb, '', '', '', '', '', 6, '地域维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'order_analysis', 'pay_type', '支付类型', 'INT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', 'pay_type_dict', 7, '支付维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'order_analysis', 'pay_channel', '支付渠道', 'VARCHAR', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', '', 8, '支付维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'order_analysis', 'channel', '分销渠道', 'VARCHAR', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', '', 9, '渠道维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'order_analysis', 'activity_type', '活动类型', 'INT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', '', 10, '活动维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'order_analysis', 'buyer_nick', '买家昵称', 'VARCHAR', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","LIKE"]'::jsonb, '', '', '', '', '', 11, '用户维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'order_analysis', 'item_title', '商品标题', 'VARCHAR', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","LIKE"]'::jsonb, '', '', '', '', '', 12, '商品维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), + +-- 指标字段 +(1, 'kuaishou_ecommerce', 'order_analysis', 'order_count', '订单数', 'INT', 'STRING', 'INDICATOR', true, false, true, true, 'COUNT', '["COUNT","SUM"]'::jsonb, '[]'::jsonb, '1', 'CALCULATED', '#,##0', '单', '', 21, '订单指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'order_analysis', 'total_fee', '订单金额(分)', 'BIGINT', 'STRING', 'INDICATOR', true, true, true, true, 'SUM', '["SUM","AVG","MAX","MIN"]'::jsonb, '["=",">=","<=","BETWEEN"]'::jsonb, '', 'ORIGINAL', '', '分', '', 22, '金额指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'order_analysis', 'total_fee_yuan', '订单金额(元)', 'FLOAT', 'STRING', 'INDICATOR', true, true, true, true, 'SUM', '["SUM","AVG","MAX","MIN"]'::jsonb, '["=",">=","<=","BETWEEN"]'::jsonb, 'total_fee / 100.0', 'CALCULATED', '#,##0.00', '元', '', 23, '金额指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'order_analysis', 'express_fee', '运费(分)', 'BIGINT', 'STRING', 'INDICATOR', true, true, true, true, 'SUM', '["SUM","AVG"]'::jsonb, '["=",">=","<="]'::jsonb, '', 'ORIGINAL', '', '分', '', 24, '金额指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'order_analysis', 'express_fee_yuan', '运费(元)', 'FLOAT', 'STRING', 'INDICATOR', true, true, true, true, 'SUM', '["SUM","AVG"]'::jsonb, '["=",">=","<="]'::jsonb, 'express_fee / 100.0', 'CALCULATED', '#,##0.00', '元', '', 25, '金额指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'order_analysis', 'discount_fee', '优惠金额(分)', 'BIGINT', 'STRING', 'INDICATOR', true, true, true, true, 'SUM', '["SUM","AVG"]'::jsonb, '["=",">=","<="]'::jsonb, '', 'ORIGINAL', '', '分', '', 26, '金额指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'order_analysis', 'discount_fee_yuan', '优惠金额(元)', 'FLOAT', 'STRING', 'INDICATOR', true, true, true, true, 'SUM', '["SUM","AVG"]'::jsonb, '["=",">=","<="]'::jsonb, 'discount_fee / 100.0', 'CALCULATED', '#,##0.00', '元', '', 27, '金额指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'order_analysis', 'original_price', '原价(分)', 'BIGINT', 'STRING', 'INDICATOR', true, true, true, true, 'SUM', '["SUM","AVG"]'::jsonb, '["=",">=","<="]'::jsonb, '', 'ORIGINAL', '', '分', '', 28, '金额指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'order_analysis', 'item_num', '商品件数', 'INT', 'STRING', 'INDICATOR', true, true, true, true, 'SUM', '["SUM","AVG"]'::jsonb, '["=",">=","<="]'::jsonb, '', 'ORIGINAL', '#,##0', '件', '', 29, '商品指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'order_analysis', 'avg_order_amount', '客单价(元)', 'FLOAT', 'STRING', 'INDICATOR', true, false, true, true, 'AVG', '["AVG"]'::jsonb, '[]'::jsonb, 'total_fee / 100.0 / NULLIF(order_count, 0)', 'CALCULATED', '#,##0.00', '元', '', 30, '金额指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'order_analysis', 'discount_rate', '优惠率(%)', 'FLOAT', 'STRING', 'INDICATOR', true, false, true, true, 'AVG', '["AVG"]'::jsonb, '[]'::jsonb, 'CASE WHEN SUM(total_fee) > 0 THEN SUM(discount_fee) * 100.0 / (SUM(total_fee) + SUM(discount_fee)) ELSE 0 END', 'CALCULATED', '#,##0.00', '%', '', 31, '比率指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()) +ON CONFLICT (tenant_id, business_code, report_code, field_code) DO NOTHING; + + +-- ---------------------------------------------------------- +-- 3.2 商品分析 - 字段配置 +-- 源表: kuaishou_item_list +-- 实际列: kwaiItemId, relItemId, title, details, categoryId, categoryName, +-- price, volume, status, auditStatus, auditReason, shelfStatus, +-- itemType, createTime, updateTime, expressTemplateId, linkUrl +-- ---------------------------------------------------------- + +-- 维度字段 +INSERT INTO report_field_config (tenant_id, business_code, report_code, field_code, field_name, field_type, data_type, field_role, is_aggregatable, is_filterable, is_queryable, is_sortable, default_aggregate, valid_aggregates, filter_operators, expression, expression_type, format_pattern, unit, dict_code, sort_order, group_name, status, creator, created_at, updater, updated_at) VALUES +(1, 'kuaishou_ecommerce', 'item_analysis', 'stat_date', '统计日期', 'DATE', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=",">=","<=","BETWEEN"]'::jsonb, '', '', 'yyyy-MM-dd', '', '', 1, '时间维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'item_analysis', 'kwai_item_id', '商品ID', 'BIGINT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', '', 2, '商品维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'item_analysis', 'title', '商品标题', 'VARCHAR', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","LIKE","IN"]'::jsonb, '', '', '', '', '', 3, '商品维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'item_analysis', 'category_id', '类目ID', 'INT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', '', 4, '类目维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'item_analysis', 'category_name', '类目名称', 'VARCHAR', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN","LIKE"]'::jsonb, '', '', '', '', '', 5, '类目维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'item_analysis', 'item_type', '商品类型', 'INT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', 'item_type_dict', 6, '商品维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'item_analysis', 'shelf_status', '上下架状态', 'INT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', 'shelf_status_dict', 7, '商品维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'item_analysis', 'audit_status', '审核状态', 'INT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', '', 8, '商品维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), + +-- 指标字段 +(1, 'kuaishou_ecommerce', 'item_analysis', 'sales_volume', '销量', 'INT', 'STRING', 'INDICATOR', true, true, true, true, 'SUM', '["SUM","AVG","MAX","MIN"]'::jsonb, '["=",">=","<=","BETWEEN"]'::jsonb, '', 'ORIGINAL', '#,##0', '件', '', 21, '销售指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'item_analysis', 'price', '商品价格(分)', 'BIGINT', 'STRING', 'INDICATOR', true, true, true, true, 'AVG', '["AVG","MAX","MIN"]'::jsonb, '["=",">=","<="]'::jsonb, '', 'ORIGINAL', '', '分', '', 22, '价格指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'item_analysis', 'price_yuan', '商品价格(元)', 'FLOAT', 'STRING', 'INDICATOR', true, true, true, true, 'AVG', '["AVG","MAX","MIN"]'::jsonb, '["=",">=","<="]'::jsonb, 'price / 100.0', 'CALCULATED', '#,##0.00', '元', '', 23, '价格指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'item_analysis', 'sales_amount', '销售额(分)', 'BIGINT', 'STRING', 'INDICATOR', true, true, true, true, 'SUM', '["SUM","AVG","MAX","MIN"]'::jsonb, '["=",">=","<="]'::jsonb, '', 'ORIGINAL', '', '分', '', 24, '销售指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'item_analysis', 'sales_amount_yuan', '销售额(元)', 'FLOAT', 'STRING', 'INDICATOR', true, true, true, true, 'SUM', '["SUM","AVG","MAX","MIN"]'::jsonb, '["=",">=","<="]'::jsonb, 'sales_amount / 100.0', 'CALCULATED', '#,##0.00', '元', '', 25, '销售指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'item_analysis', 'item_count', '商品数', 'INT', 'STRING', 'INDICATOR', true, false, true, true, 'COUNT', '["COUNT"]'::jsonb, '[]'::jsonb, 'COUNT(DISTINCT kwai_item_id)', 'CALCULATED', '#,##0', '个', '', 26, '商品指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'item_analysis', 'avg_price_per_unit', '件均价(元)', 'FLOAT', 'STRING', 'INDICATOR', true, false, true, true, 'AVG', '["AVG"]'::jsonb, '[]'::jsonb, 'SUM(sales_amount) / 100.0 / NULLIF(SUM(sales_volume), 0)', 'CALCULATED', '#,##0.00', '元', '', 27, '价格指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()) +ON CONFLICT (tenant_id, business_code, report_code, field_code) DO NOTHING; + + +-- ---------------------------------------------------------- +-- 3.3 售后分析 - 字段配置 +-- 源表: kuaishou_refund_list +-- 实际列: refundId, oid, itemId, skuId, relSkuId, skuNick, handlingWay, +-- negotiateStatus, refundFee, refundReason, refundReasonDesc, +-- refundDesc, refundType, status, receiptStatus, buyerId, sellerId, +-- logisticsId, relItemId, submitTime, createTime, updateTime, +-- negotiateUpdateTime, endTime, expireTime +-- ---------------------------------------------------------- + +-- 维度字段 +INSERT INTO report_field_config (tenant_id, business_code, report_code, field_code, field_name, field_type, data_type, field_role, is_aggregatable, is_filterable, is_queryable, is_sortable, default_aggregate, valid_aggregates, filter_operators, expression, expression_type, format_pattern, unit, dict_code, sort_order, group_name, status, creator, created_at, updater, updated_at) VALUES +(1, 'kuaishou_ecommerce', 'refund_analysis', 'stat_date', '统计日期', 'DATE', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=",">=","<=","BETWEEN"]'::jsonb, '', '', 'yyyy-MM-dd', '', '', 1, '时间维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'refund_analysis', 'refund_id', '售后单ID', 'BIGINT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', '', 2, '售后维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'refund_analysis', 'refund_type', '售后类型', 'INT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', 'refund_type_dict', 3, '售后维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'refund_analysis', 'refund_reason_desc', '售后原因', 'VARCHAR', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN","LIKE"]'::jsonb, '', '', '', '', '', 4, '售后维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'refund_analysis', 'status', '售后状态', 'INT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', 'refund_status_dict', 5, '售后维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'refund_analysis', 'handling_way', '处理方式', 'INT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', '', 6, '售后维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'refund_analysis', 'negotiate_status', '协商状态', 'INT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', '', 7, '售后维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), + +-- 指标字段 +(1, 'kuaishou_ecommerce', 'refund_analysis', 'refund_count', '售后单数', 'INT', 'STRING', 'INDICATOR', true, false, true, true, 'COUNT', '["COUNT","SUM"]'::jsonb, '[]'::jsonb, '1', 'CALCULATED', '#,##0', '单', '', 21, '售后指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'refund_analysis', 'refund_fee', '退款金额(分)', 'BIGINT', 'STRING', 'INDICATOR', true, true, true, true, 'SUM', '["SUM","AVG","MAX","MIN"]'::jsonb, '["=",">=","<="]'::jsonb, '', 'ORIGINAL', '', '分', '', 22, '金额指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'refund_analysis', 'refund_fee_yuan', '退款金额(元)', 'FLOAT', 'STRING', 'INDICATOR', true, true, true, true, 'SUM', '["SUM","AVG","MAX","MIN"]'::jsonb, '["=",">=","<="]'::jsonb, 'refund_fee / 100.0', 'CALCULATED', '#,##0.00', '元', '', 23, '金额指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'refund_analysis', 'avg_refund_amount', '笔均退款(元)', 'FLOAT', 'STRING', 'INDICATOR', true, false, true, true, 'AVG', '["AVG"]'::jsonb, '[]'::jsonb, 'SUM(refund_fee) / 100.0 / NULLIF(SUM(refund_count), 0)', 'CALCULATED', '#,##0.00', '元', '', 24, '金额指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'refund_analysis', 'negotiate_count', '协商中数量', 'INT', 'STRING', 'INDICATOR', true, false, true, true, 'SUM', '["SUM"]'::jsonb, '[]'::jsonb, 'CASE WHEN negotiate_status = 1 THEN 1 ELSE 0 END', 'CALCULATED', '#,##0', '单', '', 25, '售后指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()) +ON CONFLICT (tenant_id, business_code, report_code, field_code) DO NOTHING; + + +-- ---------------------------------------------------------- +-- 3.4 分销订单分析 - 字段配置 +-- 源表: kuaishou_cps_order_list +-- 实际列: oid, distributorId, distributorName, sellerId, status, +-- settlementTime, settlementSuccessTime, refundTime, payTime, +-- expressFee, totalFee, commissionRate, estimatedIncome, +-- createTime, updateTime, platformDpRate, activityUserId, +-- activityUserNickname, investmentPromotionRate, +-- investmentPromotionAmount, buyerOpenId, settlementBizType, +-- promoterServiceInCome, promoterExcitationInCome, +-- investmentServiceInCome, investmentExcitationInCome, +-- orderChannel, activityId, itemId +-- ---------------------------------------------------------- + +-- 维度字段 +INSERT INTO report_field_config (tenant_id, business_code, report_code, field_code, field_name, field_type, data_type, field_role, is_aggregatable, is_filterable, is_queryable, is_sortable, default_aggregate, valid_aggregates, filter_operators, expression, expression_type, format_pattern, unit, dict_code, sort_order, group_name, status, creator, created_at, updater, updated_at) VALUES +(1, 'kuaishou_ecommerce', 'cps_order_analysis', 'stat_date', '统计日期', 'DATE', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=",">=","<=","BETWEEN"]'::jsonb, '', '', 'yyyy-MM-dd', '', '', 1, '时间维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'cps_order_analysis', 'oid', '订单ID', 'BIGINT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', '', 2, '订单维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'cps_order_analysis', 'status', '分销单状态', 'INT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', 'cps_status_dict', 3, '订单维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'cps_order_analysis', 'distributor_name', '分销者', 'VARCHAR', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","LIKE"]'::jsonb, '', '', '', '', '', 4, '分销维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'cps_order_analysis', 'order_channel', '出单渠道', 'VARCHAR', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', '', 5, '渠道维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'cps_order_analysis', 'settlement_biz_type', '业务类型', 'INT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', '', 6, '业务维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), + +-- 指标字段 +(1, 'kuaishou_ecommerce', 'cps_order_analysis', 'order_count', '订单数', 'INT', 'STRING', 'INDICATOR', true, false, true, true, 'COUNT', '["COUNT","SUM"]'::jsonb, '[]'::jsonb, '1', 'CALCULATED', '#,##0', '单', '', 21, '订单指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'cps_order_analysis', 'total_fee', '实付金额(分)', 'BIGINT', 'STRING', 'INDICATOR', true, true, true, true, 'SUM', '["SUM","AVG","MAX","MIN"]'::jsonb, '["=",">=","<="]'::jsonb, '', 'ORIGINAL', '', '分', '', 22, '金额指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'cps_order_analysis', 'total_fee_yuan', '实付金额(元)', 'FLOAT', 'STRING', 'INDICATOR', true, true, true, true, 'SUM', '["SUM","AVG","MAX","MIN"]'::jsonb, '["=",">=","<="]'::jsonb, 'total_fee / 100.0', 'CALCULATED', '#,##0.00', '元', '', 23, '金额指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'cps_order_analysis', 'express_fee', '运费(分)', 'BIGINT', 'STRING', 'INDICATOR', true, true, true, true, 'SUM', '["SUM","AVG"]'::jsonb, '["=",">=","<="]'::jsonb, '', 'ORIGINAL', '', '分', '', 24, '金额指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'cps_order_analysis', 'estimated_income', '分销预估收入(分)', 'BIGINT', 'STRING', 'INDICATOR', true, true, true, true, 'SUM', '["SUM","AVG","MAX"]'::jsonb, '["=",">=","<="]'::jsonb, '', 'ORIGINAL', '', '分', '', 25, '收入指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'cps_order_analysis', 'estimated_income_yuan', '分销预估收入(元)', 'FLOAT', 'STRING', 'INDICATOR', true, true, true, true, 'SUM', '["SUM","AVG","MAX"]'::jsonb, '["=",">=","<="]'::jsonb, 'estimated_income / 100.0', 'CALCULATED', '#,##0.00', '元', '', 26, '收入指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'cps_order_analysis', 'commission_rate', '分销佣金率(%)', 'BIGINT', 'STRING', 'INDICATOR', true, true, true, true, 'AVG', '["AVG","MAX","MIN"]'::jsonb, '["=",">=","<="]'::jsonb, 'commission_rate / 10.0', 'CALCULATED', '#,##0.0', '%', '', 27, '佣金指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()) +ON CONFLICT (tenant_id, business_code, report_code, field_code) DO NOTHING; + + +-- ---------------------------------------------------------- +-- 3.5 代发订单分析 - 字段配置 +-- 源表: kuaishou_dropshipping_order_list +-- 实际列: oid, dropshippingOrderCode, payTime, buyerOpenId, buyerNick, +-- sellerOpenId, sellerNick, orderStatus, orderStatusDesc, +-- refundStatus, refundStatusDesc, orderType, orderTypeDesc, +-- deliveryTime, createTime, updateTime, waybillCode, +-- expressCompanyCode, expressCompanyName, factoryCode, factoryName, +-- allocateTime, dropshippingStatus, dropshippingStatusDesc, +-- cancelAllocateTime, cancelAllocateReason, name, mobile, +-- provinceName, cityName, districtName, detailAddress +-- ---------------------------------------------------------- + +-- 维度字段 +INSERT INTO report_field_config (tenant_id, business_code, report_code, field_code, field_name, field_type, data_type, field_role, is_aggregatable, is_filterable, is_queryable, is_sortable, default_aggregate, valid_aggregates, filter_operators, expression, expression_type, format_pattern, unit, dict_code, sort_order, group_name, status, creator, created_at, updater, updated_at) VALUES +(1, 'kuaishou_ecommerce', 'dropshipping_analysis', 'stat_date', '统计日期', 'DATE', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=",">=","<=","BETWEEN"]'::jsonb, '', '', 'yyyy-MM-dd', '', '', 1, '时间维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'dropshipping_analysis', 'dropshipping_order_code', '代发订单编码', 'VARCHAR', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', '', 2, '订单维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'dropshipping_analysis', 'order_status', '订单状态', 'INT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', 'dropshipping_status_dict', 3, '订单维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'dropshipping_analysis', 'dropshipping_status', '代发状态', 'INT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', '', 4, '代发维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'dropshipping_analysis', 'factory_name', '代发厂家', 'VARCHAR', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","LIKE"]'::jsonb, '', '', '', '', '', 5, '厂家维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'dropshipping_analysis', 'express_company_name', '物流公司', 'VARCHAR', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","LIKE"]'::jsonb, '', '', '', '', '', 6, '物流维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'dropshipping_analysis', 'province_name', '省份', 'VARCHAR', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN","LIKE"]'::jsonb, '', '', '', '', '', 7, '地域维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'dropshipping_analysis', 'city_name', '城市', 'VARCHAR', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN","LIKE"]'::jsonb, '', '', '', '', '', 8, '地域维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'dropshipping_analysis', 'order_type', '订单类型', 'INT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', '', 9, '订单维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'dropshipping_analysis', 'refund_status', '售后状态', 'INT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', '', 10, '售后维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), + +-- 指标字段 +(1, 'kuaishou_ecommerce', 'dropshipping_analysis', 'order_count', '订单数', 'INT', 'STRING', 'INDICATOR', true, false, true, true, 'COUNT', '["COUNT","SUM"]'::jsonb, '[]'::jsonb, '1', 'CALCULATED', '#,##0', '单', '', 21, '订单指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'dropshipping_analysis', 'factory_count', '厂家数', 'INT', 'STRING', 'INDICATOR', true, false, true, true, 'COUNT', '["COUNT"]'::jsonb, '[]'::jsonb, 'COUNT(DISTINCT factory_code)', 'CALCULATED', '#,##0', '个', '', 22, '厂家指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()), +(1, 'kuaishou_ecommerce', 'dropshipping_analysis', 'pending_allocate_count', '待分配数', 'INT', 'STRING', 'INDICATOR', true, false, true, true, 'SUM', '["SUM"]'::jsonb, '[]'::jsonb, 'CASE WHEN dropshipping_status = 1 THEN 1 ELSE 0 END', 'CALCULATED', '#,##0', '单', '', 23, '订单指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()) +ON CONFLICT (tenant_id, business_code, report_code, field_code) DO NOTHING; + + +-- ============================================= +-- 4. 抽取配置 (report_extract_config) +-- ============================================= + +-- ---------------------------------------------------------- +-- 4.1 订单分析 - AGGREGATE 聚合抽取 +-- 源: kuaishou_order_list → 目标: kuaishou_order_stat +-- 按 createTime 转 stat_date，GROUP BY 维度字段，聚合指标 +-- ---------------------------------------------------------- +INSERT INTO report_extract_config (tenant_id, business_code, report_code, extract_code, extract_name, source_table_name, source_table_alias, target_table_name, is_enabled, extract_type, extract_mode, extract_key_field, extract_key_format, group_by_fields, filter_expression, join_configs, field_mappings, transform_rules, batch_size, status, creator, created_at, updater, updated_at) VALUES +(1, 'kuaishou_ecommerce', 'order_analysis', 'order_daily_stat', '订单每日统计抽取', + 'kuaishou_order_list', 'o', + 'kuaishou_order_stat', true, 'INCREMENTAL', 'AGGREGATE', + 'updateTime', 'yyyy-MM-dd', + '["stat_date","oid","status","province","city","district","pay_type","pay_channel","channel","activity_type","buyer_nick","item_title"]'::jsonb, + '', + '[]'::jsonb, + '[ + {"sourceField":"stat_date","targetField":"stat_date","fieldType":"DATE"}, + {"sourceField":"oid","targetField":"oid","fieldType":"BIGINT","defaultValue":0}, + {"sourceField":"status","targetField":"status","fieldType":"INT","defaultValue":0}, + {"sourceField":"province","targetField":"province","fieldType":"VARCHAR"}, + {"sourceField":"city","targetField":"city","fieldType":"VARCHAR"}, + {"sourceField":"district","targetField":"district","fieldType":"VARCHAR"}, + {"sourceField":"payType","targetField":"pay_type","fieldType":"INT","defaultValue":0}, + {"sourceField":"payChannel","targetField":"pay_channel","fieldType":"VARCHAR"}, + {"sourceField":"channel","targetField":"channel","fieldType":"VARCHAR"}, + {"sourceField":"activityType","targetField":"activity_type","fieldType":"INT","defaultValue":0}, + {"sourceField":"buyerNick","targetField":"buyer_nick","fieldType":"VARCHAR"}, + {"sourceField":"itemTitle","targetField":"item_title","fieldType":"VARCHAR"}, + {"sourceField":"oid","targetField":"order_count","fieldType":"INT","aggregateFunction":"COUNT","defaultValue":0}, + {"sourceField":"totalFee","targetField":"total_fee","fieldType":"BIGINT","aggregateFunction":"SUM","defaultValue":0}, + {"sourceField":"expressFee","targetField":"express_fee","fieldType":"BIGINT","aggregateFunction":"SUM","defaultValue":0}, + {"sourceField":"discountFee","targetField":"discount_fee","fieldType":"BIGINT","aggregateFunction":"SUM","defaultValue":0}, + {"sourceField":"originalPrice","targetField":"original_price","fieldType":"BIGINT","aggregateFunction":"SUM","defaultValue":0}, + {"sourceField":"num","targetField":"item_num","fieldType":"INT","aggregateFunction":"SUM","defaultValue":0} + ]'::jsonb, + '[ + {"ruleCode":"ts_to_date","ruleType":"FORMAT","expression":"","format":"yyyy-MM-dd","sourceField":"createTime","targetField":"stat_date"} + ]'::jsonb, + 500, 'ACTIVE', 'admin', NOW(), 'admin', NOW()) +ON CONFLICT (tenant_id, business_code, report_code, extract_code) DO NOTHING; + + +-- ---------------------------------------------------------- +-- 4.2 商品分析 - AGGREGATE 聚合抽取 +-- 源: kuaishou_item_list → 目标: kuaishou_item_stat +-- ---------------------------------------------------------- +INSERT INTO report_extract_config (tenant_id, business_code, report_code, extract_code, extract_name, source_table_name, source_table_alias, target_table_name, is_enabled, extract_type, extract_mode, extract_key_field, extract_key_format, group_by_fields, filter_expression, join_configs, field_mappings, transform_rules, batch_size, status, creator, created_at, updater, updated_at) VALUES +(1, 'kuaishou_ecommerce', 'item_analysis', 'item_daily_stat', '商品每日统计抽取', + 'kuaishou_item_list', 'i', + 'kuaishou_item_stat', true, 'INCREMENTAL', 'AGGREGATE', + 'updateTime', 'yyyy-MM-dd', + '["stat_date","kwai_item_id","title","category_id","category_name","item_type","shelf_status","audit_status"]'::jsonb, + '', + '[]'::jsonb, + '[ + {"sourceField":"stat_date","targetField":"stat_date","fieldType":"DATE"}, + {"sourceField":"kwaiItemId","targetField":"kwai_item_id","fieldType":"BIGINT","defaultValue":0}, + {"sourceField":"title","targetField":"title","fieldType":"VARCHAR"}, + {"sourceField":"categoryId","targetField":"category_id","fieldType":"INT","defaultValue":0}, + {"sourceField":"categoryName","targetField":"category_name","fieldType":"VARCHAR"}, + {"sourceField":"itemType","targetField":"item_type","fieldType":"INT","defaultValue":0}, + {"sourceField":"shelfStatus","targetField":"shelf_status","fieldType":"INT","defaultValue":0}, + {"sourceField":"auditStatus","targetField":"audit_status","fieldType":"INT","defaultValue":0}, + {"sourceField":"volume","targetField":"sales_volume","fieldType":"INT","aggregateFunction":"SUM","defaultValue":0}, + {"sourceField":"price","targetField":"price","fieldType":"BIGINT","aggregateFunction":"AVG","defaultValue":0}, + {"sourceField":"kwaiItemId","targetField":"sales_amount","fieldType":"BIGINT","aggregateFunction":"SUM","defaultValue":0, + "transformRule":{"ruleCode":"volume_times_price","ruleType":"CALCULATE","expression":"volume * price"}} + ]'::jsonb, + '[ + {"ruleCode":"ts_to_date","ruleType":"FORMAT","expression":"","format":"yyyy-MM-dd","sourceField":"createTime","targetField":"stat_date"} + ]'::jsonb, + 500, 'ACTIVE', 'admin', NOW(), 'admin', NOW()) +ON CONFLICT (tenant_id, business_code, report_code, extract_code) DO NOTHING; + + +-- ---------------------------------------------------------- +-- 4.3 售后分析 - AGGREGATE 聚合抽取 +-- 源: kuaishou_refund_list → 目标: kuaishou_refund_stat +-- ---------------------------------------------------------- +INSERT INTO report_extract_config (tenant_id, business_code, report_code, extract_code, extract_name, source_table_name, source_table_alias, target_table_name, is_enabled, extract_type, extract_mode, extract_key_field, extract_key_format, group_by_fields, filter_expression, join_configs, field_mappings, transform_rules, batch_size, status, creator, created_at, updater, updated_at) VALUES +(1, 'kuaishou_ecommerce', 'refund_analysis', 'refund_daily_stat', '售后每日统计抽取', + 'kuaishou_refund_list', 'r', + 'kuaishou_refund_stat', true, 'INCREMENTAL', 'AGGREGATE', + 'updateTime', 'yyyy-MM-dd', + '["stat_date","refund_id","refund_type","refund_reason_desc","status","handling_way","negotiate_status"]'::jsonb, + '', + '[]'::jsonb, + '[ + {"sourceField":"stat_date","targetField":"stat_date","fieldType":"DATE"}, + {"sourceField":"refundId","targetField":"refund_id","fieldType":"BIGINT","defaultValue":0}, + {"sourceField":"refundType","targetField":"refund_type","fieldType":"INT","defaultValue":0}, + {"sourceField":"refundReasonDesc","targetField":"refund_reason_desc","fieldType":"VARCHAR"}, + {"sourceField":"status","targetField":"status","fieldType":"INT","defaultValue":0}, + {"sourceField":"handlingWay","targetField":"handling_way","fieldType":"INT","defaultValue":0}, + {"sourceField":"negotiateStatus","targetField":"negotiate_status","fieldType":"INT","defaultValue":0}, + {"sourceField":"refundId","targetField":"refund_count","fieldType":"INT","aggregateFunction":"COUNT","defaultValue":0}, + {"sourceField":"refundFee","targetField":"refund_fee","fieldType":"BIGINT","aggregateFunction":"SUM","defaultValue":0} + ]'::jsonb, + '[ + {"ruleCode":"ts_to_date","ruleType":"FORMAT","expression":"","format":"yyyy-MM-dd","sourceField":"createTime","targetField":"stat_date"} + ]'::jsonb, + 500, 'ACTIVE', 'admin', NOW(), 'admin', NOW()) +ON CONFLICT (tenant_id, business_code, report_code, extract_code) DO NOTHING; + + +-- ---------------------------------------------------------- +-- 4.4 分销订单分析 - AGGREGATE 聚合抽取 +-- 源: kuaishou_cps_order_list → 目标: kuaishou_cps_order_stat +-- ---------------------------------------------------------- +INSERT INTO report_extract_config (tenant_id, business_code, report_code, extract_code, extract_name, source_table_name, source_table_alias, target_table_name, is_enabled, extract_type, extract_mode, extract_key_field, extract_key_format, group_by_fields, filter_expression, join_configs, field_mappings, transform_rules, batch_size, status, creator, created_at, updater, updated_at) VALUES +(1, 'kuaishou_ecommerce', 'cps_order_analysis', 'cps_order_daily_stat', '分销订单每日统计抽取', + 'kuaishou_cps_order_list', 'c', + 'kuaishou_cps_order_stat', true, 'INCREMENTAL', 'AGGREGATE', + 'updateTime', 'yyyy-MM-dd', + '["stat_date","oid","status","distributor_name","order_channel","settlement_biz_type"]'::jsonb, + '', + '[]'::jsonb, + '[ + {"sourceField":"stat_date","targetField":"stat_date","fieldType":"DATE"}, + {"sourceField":"oid","targetField":"oid","fieldType":"BIGINT","defaultValue":0}, + {"sourceField":"status","targetField":"status","fieldType":"INT","defaultValue":0}, + {"sourceField":"distributorName","targetField":"distributor_name","fieldType":"VARCHAR"}, + {"sourceField":"orderChannel","targetField":"order_channel","fieldType":"VARCHAR"}, + {"sourceField":"settlementBizType","targetField":"settlement_biz_type","fieldType":"INT","defaultValue":0}, + {"sourceField":"oid","targetField":"order_count","fieldType":"INT","aggregateFunction":"COUNT","defaultValue":0}, + {"sourceField":"totalFee","targetField":"total_fee","fieldType":"BIGINT","aggregateFunction":"SUM","defaultValue":0}, + {"sourceField":"expressFee","targetField":"express_fee","fieldType":"BIGINT","aggregateFunction":"SUM","defaultValue":0}, + {"sourceField":"estimatedIncome","targetField":"estimated_income","fieldType":"BIGINT","aggregateFunction":"SUM","defaultValue":0}, + {"sourceField":"commissionRate","targetField":"commission_rate","fieldType":"BIGINT","aggregateFunction":"AVG","defaultValue":0} + ]'::jsonb, + '[ + {"ruleCode":"ts_to_date","ruleType":"FORMAT","expression":"","format":"yyyy-MM-dd","sourceField":"createTime","targetField":"stat_date"} + ]'::jsonb, + 500, 'ACTIVE', 'admin', NOW(), 'admin', NOW()) +ON CONFLICT (tenant_id, business_code, report_code, extract_code) DO NOTHING; + + +-- ---------------------------------------------------------- +-- 4.5 代发订单分析 - AGGREGATE 聚合抽取 +-- 源: kuaishou_dropshipping_order_list → 目标: kuaishou_dropshipping_stat +-- ---------------------------------------------------------- +INSERT INTO report_extract_config (tenant_id, business_code, report_code, extract_code, extract_name, source_table_name, source_table_alias, target_table_name, is_enabled, extract_type, extract_mode, extract_key_field, extract_key_format, group_by_fields, filter_expression, join_configs, field_mappings, transform_rules, batch_size, status, creator, created_at, updater, updated_at) VALUES +(1, 'kuaishou_ecommerce', 'dropshipping_analysis', 'dropshipping_daily_stat', '代发订单每日统计抽取', + 'kuaishou_dropshipping_order_list', 'd', + 'kuaishou_dropshipping_stat', true, 'INCREMENTAL', 'AGGREGATE', + 'updateTime', 'yyyy-MM-dd', + '["stat_date","dropshipping_order_code","order_status","dropshipping_status","factory_name","express_company_name","province_name","city_name","order_type","refund_status"]'::jsonb, + '', + '[]'::jsonb, + '[ + {"sourceField":"stat_date","targetField":"stat_date","fieldType":"DATE"}, + {"sourceField":"dropshippingOrderCode","targetField":"dropshipping_order_code","fieldType":"VARCHAR"}, + {"sourceField":"orderStatus","targetField":"order_status","fieldType":"INT","defaultValue":0}, + {"sourceField":"dropshippingStatus","targetField":"dropshipping_status","fieldType":"INT","defaultValue":0}, + {"sourceField":"factoryName","targetField":"factory_name","fieldType":"VARCHAR"}, + {"sourceField":"factoryCode","targetField":"factory_code","fieldType":"VARCHAR"}, + {"sourceField":"expressCompanyName","targetField":"express_company_name","fieldType":"VARCHAR"}, + {"sourceField":"provinceName","targetField":"province_name","fieldType":"VARCHAR"}, + {"sourceField":"cityName","targetField":"city_name","fieldType":"VARCHAR"}, + {"sourceField":"orderType","targetField":"order_type","fieldType":"INT","defaultValue":0}, + {"sourceField":"refundStatus","targetField":"refund_status","fieldType":"INT","defaultValue":0}, + {"sourceField":"dropshippingOrderCode","targetField":"order_count","fieldType":"INT","aggregateFunction":"COUNT","defaultValue":0} + ]'::jsonb, + '[ + {"ruleCode":"ts_to_date","ruleType":"FORMAT","expression":"","format":"yyyy-MM-dd","sourceField":"createTime","targetField":"stat_date"} + ]'::jsonb, + 500, 'ACTIVE', 'admin', NOW(), 'admin', NOW()) +ON CONFLICT (tenant_id, business_code, report_code, extract_code) DO NOTHING; + + +-- ============================================= +-- 5. 使用说明 +-- ============================================= +-- 初始化后，按以下步骤使用报表系统： +-- 1. 确保已执行 sql/seed_data_kuaishou.sql 同步源表数据 +-- 2. 在管理后台「业务管理」确认快手电商业务已加载 +-- 3. 在「报表配置」点击「自动建表」为每个报表创建统计宽表 +-- 4. 在「抽取配置」点击「执行抽取」指定 stat_date，从源表聚合数据到宽表 +-- 5. 在「数据查询」选择维度/指标/筛选条件进行查询分析 +-- +-- 关键设计原则： +-- - sourceField 严格对齐 kuaishou_xxx_list 表的实际列名（camelCase） +-- - targetField / field_code 使用 snake_case 保持项目内部命名一致 +-- - 金额字段源表存分(BIGINT)，报表提供衍生元字段(CALCULATED expression / 100.0) +-- - 抽取增量字段统一用 updateTime（毫秒时间戳 BIGINT） +-- - AGGREGATE 模式按 GROUP BY 维度聚合，非聚合字段取第一条