Provide Struct data type support for oracle plugin by vanshikaagupta22 · Pull Request #659 · data-integrations/database-plugins

vanshikaagupta22 · 2026-05-06T09:27:33Z

This change adds native support for resolving Oracle STRUCT types (Object Types) into CDAP RECORD schemas. By querying the ALL_TYPE_ATTRS metadata table, the schema builder dynamically processes complex structures with support for up to 4 levels of nesting. Internal attributes are first translated to standard SQL data types via a dedicated mapper before being converted into the final CDAP schema.
When reading the data, the implementation overrides the setField() function, providing a custom implementation to properly extract and map individual custom object attributes according to requirements.

Note: Users will require explicit EXECUTE privileges on the custom Oracle object types to successfully resolve the schema.

Test Coverage :

gemini-code-assist

Code Review

This pull request adds support for resolving Oracle STRUCT types into CDAP RECORD schemas by querying the ALL_TYPE_ATTRS metadata table. The implementation includes recursive resolution for nested structures and an updated mapping for primitive Oracle types. Feedback highlights several critical issues: the metadata query lacks an OWNER filter which could lead to incorrect schema resolution in multi-schema environments; fully qualified type names containing dots will cause IllegalArgumentException when creating CDAP records; and there are logic errors in the primitive type mapping, specifically regarding Oracle-specific type naming conventions and an invalid comparison between Java class names and SQL type strings.

# Conflicts: # oracle-plugin/src/main/java/io/cdap/plugin/oracle/OracleSourceSchemaReader.java # oracle-plugin/src/test/java/io/cdap/plugin/oracle/OracleSchemaReaderTest.java

vikasrathee-cs · 2026-07-06T13:41:15Z

+        recordBuilder.set(field.getName(), bigDecimal.longValue());
+        break;
+      case STRING:
+        recordBuilder.set(field.getName(), bigDecimal.toString());


use toPlainString()

vikasrathee-cs

LGTM

Sunish-Dahiya

Can you please add the testing report.
UTs are not following standard convention can you please fix that.
Please also attach coverage details/report.
I think we would also need to add new ITNs to cover this logic.
Since this change will require new permission, have we closed the discussion with CDF team ?

Sunish-Dahiya · 2026-07-29T09:29:36Z

+    }
+  }
+
+  private void handleOffsetDateTimeValue(OffsetDateTime offsetDateTime, Schema fieldSchema,


These types would be at top level/root level also and their handling should already exist, any specific reason we are handling record members separately?

this comment applies to all types handling

Top-level columns are read directly from the ResultSet using column positions, but STRUCT members come from struct.getAttributes() as raw Java objects where no ResultSet is available. Also, Oracle returns all STRUCT numbers as BigDecimal, dates as Timestamp/OffsetDateTime, and LOBs as Clob/Blob objects. We need this handling in populateRecordField to convert these raw objects into the correct types so that their value can be stored properly.

May be then we can move these in a separate Struct specific class

vanshikaagupta22 · 2026-08-03T09:04:02Z

Can you please add the testing report.

UTs are not following standard convention can you please fix that.

Please also attach coverage details/report.

I think we would also need to add new ITNs to cover this logic.

Since this change will require new permission, have we closed the discussion with CDF team ?

The testing report is already attached to the buganizer.
Uts are updated now.
I have attached the Test Coverage report in the Description.
I will update the ITNs in the DTS 3p CL.
Approved from CDF team

Sunish-Dahiya

The test report says coverage is around 30%, why it's so low?

Sunish-Dahiya · 2026-08-06T14:02:50Z

+    }
+  }
+
+  private void handleOffsetDateTimeValue(OffsetDateTime offsetDateTime, Schema fieldSchema,


May be then we can move these in a separate Struct specific class

… into oracle-structsupport

Sunish-Dahiya · 2026-09-30T13:16:12Z

+            handleDecimalValue((BigDecimal) attrValue, fieldSchema, recordBuilder, field);
+            return;
+        }
+        if (attrValue instanceof Timestamp) {


AI suggestion:

Struct.getAttributes() in the Oracle JDBC driver returns raw oracle.sql.* datum objects rather than java.time.OffsetDateTime or java.sql.Timestamp:

TIMESTAMP WITH TIME ZONE / TIMESTAMP WITH LOCAL TIME ZONE returns oracle.sql.TIMESTAMPTZ / oracle.sql.TIMESTAMPLTZ (which do not implement OffsetDateTime). Please extract the OffsetDateTime using reflection (offsetDateTimeValue(Connection)) as done in OracleSourceDBRecord.handleTimestampTZ.
TIMESTAMP returns oracle.sql.TIMESTAMP (which does not extend java.sql.Timestamp unless J2EE13Compliant=true).
BINARY_FLOAT / BINARY_DOUBLE can return oracle.sql.BINARY_FLOAT / oracle.sql.BINARY_DOUBLE, and UROWID returns oracle.sql.ROWID (which implements java.sql.RowId, not String).
Without converting these oracle.sql.* types first, they fall through to sourceRecord.populateRecordField and fail with UnexpectedFormatException at runtime.

can you please validate if its true?

Sunish-Dahiya · 2026-09-30T13:20:14Z

+    private static void handleTimestampValue(OracleSourceDBRecord sourceRecord, Timestamp timestamp, Schema fieldSchema,
+                                      StructuredRecord.Builder recordBuilder, Schema.Field field,
+                                      Connection connection) throws SQLException {
+        if (Schema.LogicalType.DATETIME.equals(fieldSchema.getLogicalType())) {


AI Suggestion: In Oracle, DATE attributes inside a STRUCT are returned as Timestamp. If a user overrides the nested field schema to Schema.LogicalType.DATE or Schema.Type.STRING, falling back to sourceRecord.populateRecordField will call recordBuilder.setTimestamp(...) unconditionally and fail with UnexpectedFormatException.

Please validate once

Sunish-Dahiya · 2026-09-30T13:21:14Z

+            recordBuilder.setTimestamp(field.getName(), zonedDateTime);
+        } else if (Schema.LogicalType.DATETIME.equals(fieldSchema.getLogicalType())) {
+            LocalDateTime systemLocalDateTime = offsetDateTime.atZoneSameInstant(
+                    ZoneId.systemDefault()).toLocalDateTime();


Using ZoneId.systemDefault() shifts TIMESTAMPLTZ values to the worker JVM's OS timezone rather than preserving the database/session local time (compare with OracleSourceDBRecord line 346 which uses resultSet.getTimestamp(columnIndex).toLocalDateTime()). Please avoid ZoneId.systemDefault() so output does not vary by worker machine timezone.

Sunish-Dahiya · 2026-09-30T13:25:48Z

+    if (columnTypeName != null && columnTypeName.contains(".")) {
+      return columnTypeName.substring(0, columnTypeName.lastIndexOf('.'));
+    }
+    return null;


If typeName is unqualified (e.g., "ADDRESS_TYPE" without "OWNER."), extractOwnerName returns null, and WHERE TYPE_NAME = ? AND OWNER = ? binds NULL to OWNER = ?. In Oracle SQL, OWNER = NULL always returns 0 rows, causing schema resolution to fail with No attributes found

Sunish-Dahiya · 2026-09-30T13:26:25Z

+    return null;
+  }
+
+  public Schema getSchemaMapping(ColumnMetadata metadata) throws SQLException {


does it need to be public?

Sunish-Dahiya · 2026-09-30T13:28:43Z

The test report says coverage is around 30%, why it's so low?

any update on this?

gemini-code-assist Bot reviewed May 6, 2026

View reviewed changes

Provide Struct datat type support for oracle plugin

231c6ee

vanshikaagupta22 force-pushed the oracle-structsupport branch from a88c6b7 to 231c6ee Compare May 6, 2026 14:13

Fix data reading errors

fe03c7b

vanshikaagupta22 force-pushed the oracle-structsupport branch from e6834e3 to fe03c7b Compare May 21, 2026 12:23

Merge branch 'develop' into oracle-structsupport

5ab3cd8

# Conflicts: # oracle-plugin/src/main/java/io/cdap/plugin/oracle/OracleSourceSchemaReader.java # oracle-plugin/src/test/java/io/cdap/plugin/oracle/OracleSchemaReaderTest.java

vishwasvaidya-cloudsufi force-pushed the oracle-structsupport branch 2 times, most recently from a11f5cc to af67d36 Compare June 15, 2026 10:01

Improve STRUCT support and expand data type mappings

3bfd8ba

vishwasvaidya-cloudsufi force-pushed the oracle-structsupport branch from af67d36 to 3bfd8ba Compare June 17, 2026 06:11

vanshikaagupta22 force-pushed the oracle-structsupport branch from f4ab512 to 47834bb Compare June 19, 2026 10:27

Updated schema mapping logic

219b85e

vanshikaagupta22 force-pushed the oracle-structsupport branch from 47834bb to 219b85e Compare June 19, 2026 10:53

vikasrathee-cs reviewed Jun 19, 2026

View reviewed changes

Comment thread oracle-plugin/src/main/java/io/cdap/plugin/oracle/OracleSourceDBRecord.java Outdated

vishwasvaidya-cloudsufi and others added 2 commits June 25, 2026 15:38

Refactor Oracle record parsing and remove old struct converters

a33a1d3

Fixed data type mapping

d8f92c3

vikasrathee-cs added the build label Jun 29, 2026

vikasrathee-cs reviewed Jul 1, 2026

View reviewed changes

Comment thread oracle-plugin/src/main/java/io/cdap/plugin/oracle/OracleSourceDBRecord.java Outdated

vikasrathee-cs reviewed Jul 1, 2026

View reviewed changes

Comment thread oracle-plugin/src/main/java/io/cdap/plugin/oracle/OracleSourceSchemaReader.java

vikasrathee-cs reviewed Jul 1, 2026

View reviewed changes

Comment thread oracle-plugin/src/main/java/io/cdap/plugin/oracle/OracleSourceDBRecord.java Outdated

vikasrathee-cs reviewed Jul 1, 2026

View reviewed changes

Comment thread oracle-plugin/src/main/java/io/cdap/plugin/oracle/OracleSourceDBRecord.java Outdated

vikasrathee-cs reviewed Jul 1, 2026

View reviewed changes

Comment thread oracle-plugin/src/test/java/io/cdap/plugin/oracle/OracleSchemaReaderTest.java

Resolved comments

98f58ff

vikasrathee-cs reviewed Jul 6, 2026

View reviewed changes

Comment thread oracle-plugin/src/main/java/io/cdap/plugin/oracle/OracleSourceSchemaReader.java Outdated

vikasrathee-cs reviewed Jul 6, 2026

View reviewed changes

Comment thread oracle-plugin/src/main/java/io/cdap/plugin/oracle/OracleSourceDBRecord.java Outdated

vikasrathee-cs reviewed Jul 6, 2026

View reviewed changes

Comment thread oracle-plugin/src/main/java/io/cdap/plugin/oracle/OracleSourceDBRecord.java Outdated

vanshikaagupta22 force-pushed the oracle-structsupport branch 3 times, most recently from 3455570 to 7495c92 Compare July 15, 2026 14:43

vikasrathee-cs approved these changes Jul 21, 2026

View reviewed changes

vikasrathee-cs requested review from Sunish-Dahiya and sahusanket July 21, 2026 10:12

Sunish-Dahiya requested changes Jul 27, 2026

View reviewed changes

vanshikaagupta22 force-pushed the oracle-structsupport branch 2 times, most recently from 046c0be to d24127c Compare July 27, 2026 13:25

vanshikaagupta22 requested a review from Sunish-Dahiya July 28, 2026 06:19

Sunish-Dahiya reviewed Jul 29, 2026

View reviewed changes

vanshikaagupta22 force-pushed the oracle-structsupport branch from d24127c to ca030ef Compare August 3, 2026 08:27

Minor fixes

3a6cb47

vanshikaagupta22 force-pushed the oracle-structsupport branch from ca030ef to 3a6cb47 Compare August 3, 2026 08:49

sahusanket approved these changes Aug 3, 2026

View reviewed changes

vanshikaagupta22 requested a review from Sunish-Dahiya August 3, 2026 17:34

Sunish-Dahiya reviewed Aug 6, 2026

View reviewed changes

Merge branch 'develop' of https://github.com/cloudsufi/database-plugins…

fd4635e

… into oracle-structsupport

vanshikaagupta22 force-pushed the oracle-structsupport branch from 490f15c to fd4635e Compare September 29, 2026 11:53

Sunish-Dahiya reviewed Sep 30, 2026

View reviewed changes

Conversation

vanshikaagupta22 commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vikasrathee-cs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Sunish-Dahiya left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vanshikaagupta22 commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Sunish-Dahiya left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Sunish-Dahiya commented Sep 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

vanshikaagupta22 commented May 6, 2026 •

edited

Loading

vanshikaagupta22 commented Aug 3, 2026 •

edited

Loading