Tag Archives: Database

Using a terminal emulator on Mac


Dumb title for a blog post? No, not really I guess…

ZOC Terminal emulatorI have been using a terminal emulator, basically ever since I got away from the VT100 terminal:

  • ICE.TCP Pro
  • KEAVT
  • Reflection ‘X’

And a few other obscure applications that I cannot even recall anymore.
Currently, and over the last 6 to 8 years, I have been using ZOC.

The background of this story is: In the beginning these were the first DOS PC’s and later some Windows based machines that needed to interface with (in my case) VAX VMS, and later with the other UNIXes.

But why use a terminal emulator on MAC, for crying out loud? I hear you think… OS X is a Unix, so it should be all native, right?
Wrong! Well, kind of…

There are so many small (and bigger) differences when using various systems that it pays off to have a program that allows you to tune into these differences. Nothing more annoying than a backspace key that does not work or key-combinations that act different than you would expect.
This is even more true when you work with a mix of different operating systems, Solaris, HP UX, Oracle Linux, perhaps even some IBM OS’s.
And for when you would like to have further tune-ability of you toolkit, ranging from colors, to sizes, from fonts to layout.  Frivolities? Perhaps, but if you spent a lot of your time everyday in such a  tool, it does make a difference.

Printer terminal, also a terminal emulatorMore importantly are configurable logging, for documentation and troubleshooting. You can regard this as the modern variation to the old school print terminal (who can remember those?)
Configuring transfer types, modem and commutation settings and keeping these organized. As well as password storage and administrative support.

Well, basically, this is why I use a terminal emulator on my MAC!
And I think I found a valid tool in ZOC, by Markus Schmidt. Please check it out ZOC

Well, I hope you get to enjoy your terminal work as much as I do!


Register redo-log manually with Divisit Replicate

For those of you who haven’t been working with on-line data replication; in short, it is a way to copy data from a source database to a target database and do this on-line (both databases are active) and do it near-real-time.
This means that when you enter data in you source database, you can immediately query it from your target database. This makes on-line data replication ideal for numerous tasks, like moving and / or upgrading your database while it is being used, with almost no downtime at all.

This tale is of an actual project that I conducted. I used Dbvisit Replicate as my tool of choice.

dbvisit-replicate-logical-replication-made-easy-18-638Dbvisit Replicate can use a so-called FETCHER process to act as the “long-arm” for the MINE process. Mining extracts the information from the redo-log files, but, in specific situations, this can be too much of an overhead for the source database server. By moving the MINE to a proxy server, this overhead can be significantly reduced.

In some cases it can be useful to manually transfer redo-log files to the mining stage directory of Dbvisit.
I came across this requirement when catching up a lot of redo from a RAC database. In this case, the RAC cluster creates two streams of redo. When starting the replication processes, the first thread is transferred by FETCHER from the source server to the proxy, before the second thread is transferred. This means mining will pause until the second thread successful delivers the first redo-log file of the second thread. The redo-log information from the second stream is necessary to create consistent and chronologically ordered SQL-statements for the target database. In effect, the SCN’s from first redo-log information of the first stream need to line up with the SCN’s of the second redo-log information.

In this case, this meant having to wait a day or more before mining can start. This is why I decided to copy a number of redo-log files from the source server to the proxy server, where the MINE process is running, manually.
After the copy, the files need to be registered with in the dbvrep-repository. Without this information, the MINE process has no knowledge of the files that are present and about what their contents are.

The update is an easy insert statement, but it should be handled with care, as this needs to be quite precise and it needs a bit of specific information about the redo-log files being added.
You can use the following insert statement to register the files:

insert into dbvrp.dbrsmine_redo_log_history
       (
       ddc_id
     , mine_process_name
     , sequence
     , thread
     , resetlogs_id
     , first_scn
     , next_scn
     , online_name
     , arch_name
     , read_count
     , from_fetcher
     , last_mine_start
     , last_mine_end
     , create_date
     , last_change_date
       )
values
       (
       1
     , ‘MINE’
     , 128779 -- sequence number of the copied file;
     , 2 -- assuming you are updating this thread.
     , 804864915 -- the reset-logs id from the redo-log file
     , 199910296688 -- the first scn from the redo-log file
     , 199911476897 -- the next scn from the redo-log file
     , null
     , ‘/u01/app/oracle/some-big-storage/dbvrep-mine/mine-stage/thread_2_seq_128719.1485.804864915’
       -- full path and name of the file
     , 0
     , ‘Y'
     , null
     , null
     , sysdate
     , sysdate
       )
;

And you can get the information you need about the files here:

select lh.sequence#
     , di.resetlogs_id
     , lh.first_change#
     , lh.next_change#
  from v$log_history lh
 inner join v$database_incarnation di
 using (resetlogs_change#)
 where sequence# = 128779
;

After registering the first file for the second thread, in the Replicate-console, you can watch the MINE process kick off. This process will then again halt after the first file of the second stream is processed in parallel with the first file of the first stream.

Schermafbeelding 2015-05-31 om 21.23.11

I kept adding files until the FETCHER process was able to take over, or you could do this until you test-case or PoC is over.

Setting up SQL Instant Client on MAC

In doing more work directly from my Macbook Air, I ran into a situation where native connectivity to an Oracle environment was needed.Connectivity over Oracle Instant Client
From experience I have always been a big fan of the Full Oracle client, just because it comes with a lot of tools and utilities for troubleshooting, which makes the actual experience a bit more pleasant.
Looking & asking around, though, I learned fairly quickly that this client is just not available for Mac OSX… Thanks to Osama Mustafa for confirming.

So, a fact, although quite a number of IT pro’s are working with Mac!

This leaves no other choise than to divert to the Oracle Instant Client 11, which then, indeed, is just an 11g Instant Client (11.2.0.4)!
It would humor me if Oracle were to bring out a 12c Full Client for Mac, as well as an instant client, if someone would so desire.
To have some more tooling around the client, I downloaded all the packages including at least SQL*Plus.

Though the install process is relatively straight forward (download the archives and unzip them in place) getting SQL*Plus to actually run is a somewhat different ballgame!
As usual, when you start a tool, you’re bombarded by messages about unfound dynamic libraries. This set me (very briefly) on a path to place these files where they were expected on my Mac.

In a place like:

/ade/b/2649109290/oracle/sqlplus/lib

for instance, you would need to place a number of these libraries.
This leaves you with the option to populate your system with all these specific libraries, which is of course just fine, but not my choice (think of the mess in ever having to clean up) and especially not when it can be avoided.

A quick search pointed me to this excellent blogpost by Casey Lucas about this exact same issue. With a tool called ‘otool’* applied as suggested, I am now able to run SQL*Plus natively on my Mac without error messages.

* otool – object file displaying tool
If you need it, call it from the command line. It will install this and other development tools on your Mac.

That is nice, but it’s just only over halfway there.

manneke stopt de stekker erin

 

Now I want something where I can just run:
sqlplus <username>@<database>
without intricate connect-strings.

 

 

This leaves one minor “hack”, or rather “edit” required, your .bash_profile needs a bit of a path addition and an environment setting:
alias ll="ls -l"
export TNS_ADMIN=/Applications/instantclient_11_2
export PATH=/Applications/instantclient_11_2:$PATH

Note: the alias was already in there 😉

To top it off, I created a small tnsnames.ora in the directory with the instant client (keeping all related files neatly tucked away together)

xesource =
(DESCRIPTION =
(ADDRESS =
(PROTOCOL = TCP)
(HOST = 192.168.56.66)
(PORT = 1521)
)
(CONNECT_DATA =
(SERVER = DEDICATED)
(SID = xe)
)
)

And voila, goal acquired.

sqlplus usera@xesource
Never specify the password on the command line. Not only will it be shown, it will also be sent (most probably) unencrypted over SQL*Net

OUGN15, The “boat conference” revisited

Jan at shipsport
Reflections on OUGN

Sometimes things in life can change quickly! It is only two years ago that I came to Oslo for the first time to join the Scandinavian Oracle crew on a boat trip to Kiel.
At that time I had never actually participated in this kind of experience and I wasn’t into presenting either. Together with my good friend Philippe Fierens I discovered a whole new world back then. You could have read about these experiences in some blogpost, but this was lost in the move to my own site, sorry!

And this trip couldn’t have been more different though! With three presentations accepted the two days at sea will be a reunion with the friends I made over the last years, as well as a way to contribute to one of the most tight knit tech communities I know. And this will be in a scene that I remember vividly from being a newbie… And this is somewhat strange, believe me

After a quick and pleasant flight I touched down in Oslo, flying from Amsterdam with a decent sized crew of Dutch Oracle enthusiasts, including my good friends Patrick Barel and Alex Nuijten. Waiting in the Oslo airport for Luís Marques, I catches up with Gurcan Orhan, which was a great surprise.
Later that day we found ourselves in the Oslo harbor for the speakers dinner. You can imagine the collective amount of Oracle knowledge packed into that one restaurant!

frits
Enkitek’s Frits Hoogland on Ansible

After a somewhat restless night we arrived, on Thursday morning, at the ship Color Fantasy with the Heli Helskyaho-company, just in time for the keynotes. It was good to see Mark Rittman and James Morle made it on board too. Especially as James was up for the delivery of version 2.0 of his vibrant keynote! Next we proceeded to bring our luggage to our cabins and grab a spot of lunch on the exhibition floor down in the belly of the ship. The setup of the exhibition was quite nice and gave a good opportunity to mix and mingle.
The afternoon was spent on sessions, where I visited Frits Hoogland with the Ansible talk, and preparation for my own session at 18:00. This is the last run of this APEX presentation, as I have retired it after OUGN15. The slides will be archived here.
After finalizing the preparation for the third edition of the Standard Edition Round Table (aka “slide polishing”) with the #orclSERT team, comprising of Ann Sjökvist, Philippe and myself, it was time for the souree and for diner in the grand restaurant on board. It has been a good first day!

Diner
Dinner with the international crew on board the Color Fantasy.
Gin-tonic
Warm reception at Kiel port.

The second day of OUGN15 started with a multitude of sessions including the third edition of the Oracle Standard Edition Round Table, which was actually quite busy and interactive. We had some good discussions, and that at 09:00, so thank you, everybody.
Of course, as was declared a tradition, Björn Rost was present in the Kiel harbor. With the famous “Basil smash Gin & tonic” and sandwiches we were welcomed on German soil.
My afternoon comprised 3 sessions, starting with my own called “Okay, and now my database server crashed…” which was quite nicely received. Next Alex Nuijten on 12c new features for developer, topped off with Tim Gorman who taught us to be CSI people, in finding issues in the database.
After an enjoyable evening in the various bars and discotheques of the ship we retired the official part of the Oracle User Group Norway Vårseminar 2015, thanking the board and of course especial Øyvind Isene, for their hard work.

If you want to catch up further on the unconference communications surrounding this event, please do checkout the Twitter hashtag #OUGN15. This will also include a great set of snapshots and pictures taken along the way…

Oslo, until the next time!

Oracle on OpenVMS – revival

Can it be true?

Will there be Oracle on OpenVMS again? Meaning the “regular” (sorry) Oracle (12c) RDBMS on a revived VMS?

As many who have ever lived on OpenVMS have always known:

OpenVMS will never die!

OpenVMS can never die, becOpenVMSause it is still running way to many hidden, hyper-mission-critical environments.
The fact that these environments are hidden, combined with the fact nobody ever spent any marketing budget on OpenVMS at all, created a super solution nobody knows about. And you cannot love what you don’t know.

A lot has been happening around this tormented operating system. OpenVMS is indefinitely bound to Digital Equipment Corporation (DEC) which was acquired by Compaq in June of 1998 and then merged into Hewlett Packard in May 2002.

DIGITAL-logo
Personally I have lost access to OpenVMS, and Oracle op OpenVMS, around 1995, when these systems were replace by HP Unix. I never fully recovered 😉

Until a few years ago I was introduced to a Alpha emulator, which creates a virtual machine with (obviously) an Alpha processor, which allows you to run OpenVMS. This was one step closer (back) to Oracle on OpenVMS.

Recently (like the day before yesterday, recently) I learned a number of new things! One is that the ongoing development OpenVMS will be take up by VMS Software Inc. (VSI).
But, more importantly, they will be creating new versions for mainstream hardware (such as x86)!! Wilm Boerhout of VX Company wrote an announcement about this not too long ago (article in Dutch)!

And now these rumors…

A porting of Oracle on OpenVMS!

Will we once again see the day that systems just won’t go down? Oracle environments with an uptime with like a dozen or two ‘nines’ behind the decimal-mark? Wouldn’t that be something?
Your own VMS server running an Oracle database with Oracle Application Express (APEX)? Wouldn’t that be something else? High time to clear some of your calendar and get (re)acquainted with this super OS!

A very special “thank you” goes to my dear friend Gerrit WoertmanOpenVMS Ambassador, who never seized to remain a link to the VMS World!!

If you are from The Netherlands, please also join Interexperience, to stay close to the game.

Live free or die

Big Data: Hadoop and Oracle technologies explained

MarkRittmanUnder the title “Hadoop and Oracle technologies on BI projects” Mark Rittman flew to The Netherlands on the 14th of July to visit the Oracle Usergroup Holland.

As I had obviously heard a lot about Hadoop, I never really did anything further with it and left it to a synaptic link to Gwen Shapira. This lack of action created a kind of threshold in the understanding of the technology. When I heard about this session I realized this would be the moment to take a step further. It turned out the be the  first real talk that puts “Big Data” in the perspective it needs to be consumable and realistic.

In these current times where “The Internet of Things”, more and more social media and ever further digitization we are heading to a Big Data Disruption. This is both a conceptual as a very real thing if you take a moment to think about it. According to real world experience it is also not something “which will once be”, it is something which is actually here today!

On the technical side of thhadoopings, data is captured in something that is called a “data reservoir” (or “data lake” or “data dump (yard)”). Compared with “regular” data storage, you can conclude that data-governance, or a data-structure, in a Big Data system is applied later  We are used to apply this structure, this governance, beforehand, by applying data definition. Using Hadoop in combination with noSQL give you “schema on read” capabilities making quering of the Hadoop data reservoir possible.

Adding this structure later is harder! This leads to the following:

  • Data is much easier to get into Hadoop then into a star-schema
  • Data is much easier to get out of a star-schema then out of Hadoop

This could be one of the essential things to consider when thinking about engaging in a Big Data project!

As Tanel Poder concluded: “High value, high density data will remain in the Oracle database” which I think is a very true conclusion. In the end, the high value conclusions (or the engineering of Big Data results) will also happen within the Oracle database.

On the horizon is “Oracle Big Data Discovery” which will help with the time consuming and tedious work of sorting and interpreting raw data in the data reservoir. The use of ‘R’, as the data exploration tool of duty, is expected to be replaced by this discovery tooling, over time…

To sum up the concept of the first half of the presentation, to my taste:

  • Hadoop changes business
  • NoSQL scales business
  • Oracle runs business

It takes eons to list all names of the Buddha” nicely sums up the number of different applications that make up and are needed to execute a successful Big Data project.
Plus, “You’d better keep the 13 rules for relational databases close at hand“!

presentation

Part two of the evening was spent on mapping these concepts on actually tools, disclosing data through Hadoop to Oracle SQL and making actual use of Big Data. The exercise was completed by demos and illustrated by screenshots from the slides (link below).
A special word of warning goes out to the security aspect of Big Data, which is something to really pay close attention to. Kerberos authentication and apache Sentry are imperative things to implement in your Big Data environment.

All in all, this evening turned out to be 110% more informative and necessary as I expected when I embarked on the journey to Utrecht! Thank you for sharing, Mark!

Thanks to Piet de Visser for the nice quotes! And a great “hi there” to Klaas-jan Jongsma, René Kuipers and Marti Koppelmans.

If you want to work with Big Data on your Smal(ler) Device, please download the Big data light VM from OTN.

The link to the slides for anyone who wants to review the “extended remix”!

DOAG 2014, Nüremberg visited

Traveling to Nuremberg, anticipating three days of Oracle submersion. There are so many speaker heading over there it cannot be anything but successful.
This will be the first conference I will attend after being accredited as Oracle ACE Associate which, for me, makes it again a little more special.TurboProp
The first surprise, though, was just that. Arriving, by bus, at the boarding-location, there was a Bombardier DASH8-Q400 waiting, which turned out to be a turbo-prop aircraft. Okay, I jumped from a Cessna Caravan twin engine turbo-prop before, but this was still a first. As I am writing these lines, we’re descending upon Nüremberg.

On the first day of the conference, which started with a beautiful but rainy morning stroll to the conference center, The action started to really kick in from about 12:00 with the first session of my good friend Peter Raganitsch, talking about the 10 worst practices in APEX. A refresingh way of looking at software development by focussing on how to do it wrong!
The day ended with one of the “most pleasantly unorganized sessions” of the conference, where Johannes Ahrends and Philppe Fierens joined me on stage for the Standard Edition Round Table, #DOAG14-edition.

The second day was full of sessions, and I vistited Joel Kallman “APEX fast=true”, discussing the the knowledge needed to do serious application development on APEX, creating #DBADev. And, off course, the sharp presentation of my friend Franck Pachot about interpreting AWR-reports!
At 17:00 it was time for my third event, the “Electronic Patients Records system based on Oracle APEX” talk, which had quite a good turnout.
GatheringThe day ended with a super-cool meet-up with Mia Urman, Lonneke Dikmans AND Brynn Llewellyn… And later on we had a real nice depiction of #DBADev 2.0, involving Joel Kallman, Philippe Fierens, Illoon Ellen and myself.

Gathering

The third and last day of the conference was spend executing #RepAttack. This session concluded 3 full days of hands-on hacking with cool software and getting a feel of some of the new stuff.

RepAttack

A few of the cool new meetings (which we’ve dubbed the e-people to real people conversion by IRL) involved:

Thank you, DOAG, for a superb conference. I thoroughly enjoyed it. To all the Oracle aficionados, until next time!!

Upgrading 11.1 to 11.2 and the time it takes

As many know, Oracle 12 has not reached each and every corner of many production sites. Okay, running Oracle 7 or 8 is becoming tricky…
Moving from anywhere to Oracle 11.2.0.4.0 is still a valid action as it is still the latest and rock-solid stable release out there.

I wanted to share what I ran into, upgrading 11.1.0.7.0 to 11.2.0.4.0 with in-place upgrades (catupgr.sql). As 11.1 to 11.2 is a relatively small step, in quite a few instances we have chosen an in-place upgrade over ‘ye old fashioned export/import with the main reason; saving time and reducing chances on error as you factually stay in the same database.

On the topic of “saving time”… this is what I found…

At some point during the upgrade, tailing the log, I noticed that an unreasonable and unexplainable amount of time (50% of the entire upgrade duration in my example) was spent on just this one statement:

— revoke grant with grant option privs

Note: Contact me for the actual statement…

This got me puzzled to such a state, and since The Internet didn’t hold any of the answers, I decided to turn to MOS and raise an SR.

And after just a few messages to and fro, the issue was found!

The time this bit of program spends, is spend on ORDIM and SDO, better known as Oracle Intermedia and Oracle Spatial. Bringing me to the task (which I knew that answer on already, as a matter of fact) of finding out if both of these technologies were used, and that was easy enough:

connect / as sysdba
— ORDIM
select owner, table_name, column_name
from dba_tab_cols
where data_type in (‘ORDAUDIO’,’ORDDOC’,’ORDIMAGE’,’ORDSOURCE’,’ORDVIDEO’)
order by 1,2,3;
— SDO
select owner,index_name from dba_indexes
where ityp_name = ‘SPATIAL_INDEX’;
select owner, table_name, column_name
from dba_tab_columns
where data_type = ‘SDO_GEOMETRY’
and owner != ‘MDSYS’
order by 1,2,3;

Which in my case returned “no rows selected” as expected.

With this knowledge “in pocket”, it was a matter of removing the unwanted matter

  1. Deinstall Oracle Spatial (SDO) following the steps listed in Note.179472.1.
  2. Deinstall ORDIM per $ORACLE_HOME/ord/im/admin/README.txt, see Note.337415.1.

And do the upgrade in half the time.

Well, hope this helps to save you some time.

Can you boost your Oracle database performance on HP-UX for free?

Database performance, as is true with all performance related matters, has to do with resources.

This story specifically focuses on a real life experience with Oracle database performance on HP Unix running on Intel integrity CPU’s like these:

CPUinfo

The issue with installation is the hyper-threading aka. the use of the logical processors.
When the server is booted and is running, you can do basic performance review with a default tool like top.

lcpu_attr=0

In this exact case the server is running fine and there is no need to investigate further. But, in cases where there were performance issues, it would be a good idea to be aware of the numbering of the CPU’s in this overview (0, 2). This numbering suggests there would also be a ‘1’ and, where there would be a ‘1’ there would probably also be a ‘2’…

Yes there is and it is called ‘lcpu_attr’. A HP Unix kernel parameter which is, to my taste, a bit odd, not well known or well documented…

lcpu_attr (Tunable Kernel Parameter)

When turned on, lcpu_attr activates the logical CPU’s immediately. When you run top again, this is what it’ll look like (immediately)

lcpu_attr=1

Okay! Great… but… there are some catches.
This parameter lcpu_attr is a dynamically tunable kernel parameter but… it’ll crash your databases. So you will need a minimum of planned downtime for this action.

Also, you can set hyper-threading on in the EFI boot-loader.
But then you should be aware of this!

In the end, in this real-life story, we helped the situation advance by just doing:

1. stopping Oracle database(s)
2. kctune lcpu_attr=1
3. starting Oracle databas(s)

All in all, it could be not difficult to boost your Oracle database performance on HP-UX for free!

Thanks to my good friend Gerard van der Kooij for finding the final link!

Increasing the reach of your SE database license

Imagine the following situation…

Since a few years your business has been investing in centralizing valuable business information. After some research in the market you have found the Oracle database to be the best fit for your requirements.
Using the free Oracle Application Express (APEX) framework, helping you to rapidly develop the web-applications needed to support both internal and external users, was a premium. Making this installation available based on the Oracle Standard Edition One database, you have created this solution against the lowest possible investment!

As many great projects go, the use and the number of APEX applications is growing. With the addition of ready to use applications to inspire you, many cool plug-ins to ever increase the usability and integration possibilities you get caught up in the data growth dogma!
With an ever increasing user population and expansion of data-reporting for ever faster business reporting your initial system is starting to fail, showing ever more frequent performance lags or system unavailability. These problems form a risk for your business, a risk you need to eliminate as soon as possible!
The standard advise here would be to upgrade your environment, the standard advise here would be to upgrade to a bigger machine and to an Enterprise Edition database. This is what your investment would be then…

  • Medium Oracle Sun Server X2-4 with 4 x 10 core CPU’s at € 42,500
  • (40 cores x 0,5 core-factor **) 20 Oracle Database Enterprise Edition licenses               at € 914,800

Without rendering your application infrastructure worthless by the required investment, a more reasonable step would be to migrate to Oracle Database Standard Edition.

  • Medium Oracle Sun Server X2-4 with 4 x 10 core CPU’s at € 42,500
  • 4 Oracle Database Standard Edition licenses at € 67,400

Still requiring a total investment of more than a hundred thousand Euro and leaving you with the old server and licenses to be decommissioned.

In many implementations, not data entry but data-mining or information aggregation are the costly processes. So probably this will be true in this situation too. With a little investigation it is possible to separate a number of functions that will only query data and not necessarily modify data. Especially in this situation you can also increase your application performance by moving these specific processes to a new environment.

But… how…

The information in the new environment needs to be real-time consistent with the “production” or primary environment. Here we introduce a real-time data replication solution like Dbvisit Replicate which will create just this real-time consistent query environment for you! This makes for the following investment:

  • Medium Oracle Sun Server X4-2 with 2 x 8 core CPU’s at € 19,500
  • 2 Oracle Database Standard Edition One licenses at € 11,200
  • 4 Dbvisit Replicate XTD at € 16,180

With this installation you add another € 50 k. of licensing in stead of € 100 k. with the Standard Edition migration. With this choice, you separate your time-critical data-entry process from the query environment, making sure a mis-fired query will not influence the availability of your data-entry process environment, which is a cool extra advantage!

* All prices are based on list-prices, excluding VAT and including 1 year of support.
** Based on the Oracle Processor Core Factor Table.